Search and the web

Each parser reads one kind of page and returns the site’s own data. Send the page’s URL to scrape with a json format that names the parser. The data comes back in data.json.

curl -X POST https://api.stophy.dev/v1/scrape \
  -H "Authorization: Bearer $STOPHY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://scholar.google.com/scholar?q=attention+is+all+you+need","formats":[{"type":"json","parser":"google-scholar"}]}'

const response = await fetch("https://api.stophy.dev/v1/scrape", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.STOPHY_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://scholar.google.com/scholar?q=attention+is+all+you+need",
    formats: [{ type: "json", parser: "google-scholar" }],
  }),
});
console.log(await response.json());

import os

import requests

response = requests.post(
    "https://api.stophy.dev/v1/scrape",
    headers={"Authorization": f"Bearer {os.environ['STOPHY_API_KEY']}"},
    json={
        "url": "https://scholar.google.com/scholar?q=attention+is+all+you+need",
        "formats": [{"type": "json", "parser": "google-scholar"}],
    },
)
print(response.json())

Send a parser’s options in the same format, like {"type": "json", "parser": "google-scholar", "options": {...}}. See Parsers for how they work.

Parser	Reads	Price
`google-scholar`	A Google Scholar results page with a search in q or a cites id.	1 credit per call.
`google-patents`	A Google Patents search page with q, inventor or assignee.	1 credit per call.
`google-trends`	A Google Trends explore page with search terms in q, or the trending now page.	1 credit per call.
`emails`	Any public web page.	1 credit per call.
`phone-numbers`	Any public web page.	1 credit per call.

google-scholar

A Google Scholar results page with a search in q or a cites id.

Example page: https://scholar.google.com/scholar?q=attention+is+all+you+need
Returns: title, url, format, authors, publication, year.
Options: citesId (A result’s citesId: lists the works that cite it, like Scholar’s Cited by), yearFrom (Scholar’s own Custom range: from this year), yearTo (Scholar’s own Custom range: up to this year), sort (Scholar’s own sort. Newest lists recent additions first), type (Scholar’s own Any type, Review articles, or Case law), includePatents, includeCitations (Scholar’s own include citations: works known only from citations), page (Page number, starting at 1).
Price: 1 credit per call.

Example response

{
  "results": [
    {
      "title": "Attention is all you need",
      "url": "https://proceedings.neurips.cc/paper_files/paper/7181-attention-is",
      "format": "pdf",
      "authors": [
        "A Vaswani",
        "N Shazeer",
        "N Parmar"
      ],
      "publication": "Advances in neural …",
      "year": 2017,
      "publisher": "proceedings.neurips.cc",
      "snippet": "… to attend to all positions in the decoder up to and including that position. We need to prevent … We implement this…",
      "citations": 279946,
      "citesId": "2960712678066186980",
      "versions": 26,
      "pdfUrl": "https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf?src_trk=em66c5166…",
      "position": 1
    }
  ],
  "page": 1
}

google-patents

A Google Patents search page with q, inventor or assignee.

Example page: https://patents.google.com/?q=solid+state+battery&assignee=Toyota
Returns: patentId, title, snippet, assignee, inventor, priorityDate.
Options: inventor, office (Google Patents’ own Patent office, like US, EP, WO, CN or JP), language, dateType (Which date before and after compare: Google Patents’ own date choice), after, before, status, type, litigation (Google Patents’ own Litigation: has related litigation, or no known), sort, page (Page number, starting at 1).
Price: 1 credit per call. Works without an API key.

Example response

{
  "results": [
    {
      "patentId": "US11018374B2",
      "title": "All-solid-state battery",
      "snippet": "5. The all-solid-state battery according to claim 1 , wherein, when the total volume of the anode layer is determined…",
      "assignee": "Toyota Jidosha Kabushiki Kaisha",
      "inventor": "Satoshi Mizutani",
      "priorityDate": "2018-04-03",
      "filingDate": "2019-03-14",
      "grantDate": "2021-05-25",
      "publicationDate": "2021-05-25",
      "language": "en",
      "patentUrl": "https://patents.google.com/patent/US11018374B2/en",
      "pdfUrl": "https://patentimages.storage.googleapis.com/a9/f6/23/69fb5afe31ad17/US11018374.pdf",
      "thumbnailUrl": "https://patentimages.storage.googleapis.com/d1/48/46/5134db0b904d57/US11018374-20210525-D00000.png",
      "position": 1
    }
  ],
  "page": 1
}

google-trends

A Google Trends explore page with search terms in q, or the trending now page. One term gives its related searches and several terms give their interest over time. Set options.view to related, interest or regions to pick that answer for the terms.

Example page: https://trends.google.com/trends/explore?q=bitcoin&geo=US&date=today%2012-m
Returns: query, value.
Options: searchType (Google Trends’ own search type: web, images, news, shopping or youtube), category (Google Trends’ own top-level category), resolution, country (ISO 3166-1 alpha-2 country code, e.g. us. Any case is accepted), time (Google Trends’ own Trending now time range), status (Google Trends’ own Trend status: all, or only trends still rising), sort (Google Trends’ own sort).
Price: 1 credit per call.

Example response

{
  "results": [
    {
      "query": "bitcoin price",
      "value": 100
    }
  ],
  "rising": [
    {
      "query": "spcx",
      "increasePercent": null,
      "isBreakout": true
    }
  ]
}

emails

Any public web page.

Example page: https://www.hetzner.com/legal/legal-notice/
Returns: email, pages.
Price: 1 credit per call.

Example response

{
  "url": "https://www.hetzner.com/legal/legal-notice/",
  "emails": [
    {
      "email": "name@example.com",
      "pages": [
        "https://www.hetzner.com/legal/legal-notice/",
        "https://www.hetzner.com/unternehmen/ueber-uns/"
      ]
    }
  ]
}

phone-numbers

Any public web page.

Example page: https://www.hetzner.com/legal/legal-notice/
Returns: phone, pages.
Options: country (Two-letter country code for numbers written without a country prefix, such as DE. Without it the country comes from the page: its numbers with a prefix, its address and its domain ending).
Price: 1 credit per call.

Example response

{
  "url": "https://www.hetzner.com/legal/legal-notice/",
  "phones": [
    {
      "phone": "+1 555 0100",
      "pages": [
        "https://www.hetzner.com/legal/legal-notice/",
        "https://www.hetzner.com/support/",
        "https://www.hetzner.com/unternehmen/ueber-uns/"
      ]
    }
  ]
}

Get started

Parsers

Guides

Plans and billing

Search and the web

google-scholar

google-patents

google-trends

emails

phone-numbers

​google-scholar

​google-patents

​google-trends

​emails

​phone-numbers

google-scholar

google-patents

google-trends

emails

phone-numbers