• $ ls contents

  • ..
  • Controlling the browser
  • • Capabilities
    • Usage
    • Usage from workbench

For info on how to use the browser non-technically, check out our product walkthroughs or this video showing how the browser can help quickly "pluck" data from a website.

Capabilities

Let's suppose we're interested in programmatically getting Google search results without using their API. You'd have two options for going about this:

• Craft the search results URL to, in one HTTP request, get the page containing links of interest
• Navigate to the Google home page, enter a query, hit search, and then extract from the resulting page

Going with the former means you're eventually constrained in agility with respect adding new websites; your limiting factor becomes how quickly you invent clever tricks for getting the new data you're looking for. However, the latter describes how you would get the data if you were to do it by hand. Shown below is a finished example of an script that goes along the approach described in the second bullet point above. In LSD, browser operations are represented in an semi-instruction language inside your query. Let's look at what each part is doing.

search_input <| textarea[name="q"] |
google <| https://google.com |
query_to_try <| "What is LSD.so?" |
search_button <| form center:nth-child(2) input[value="Google Search"] |
search_item_container <| div#search a |
search_item_link <| div#search a@href |

FROM google
|> ENTER INTO search_input query_to_try
|> CLICK ON search_button
|> GROUP BY search_item_container
|> SELECT query_to_try, search_item_link

The lines at the top of the SQL script are variables that help the rest of the script be easier to understand and edit.

search_input <| textarea[name="q"] |
google <| https://google.com |
query_to_try <| "What is LSD.so?" |
search_button <| form center:nth-child(2) input[value="Google Search"] |
search_item_container <| div#search a |
search_item_link <| div#search a@href |

FROM google
|> ENTER INTO search_input query_to_try
|> CLICK ON search_button
|> GROUP BY search_item_container
|> SELECT query_to_try, search_item_link

Next, we state we are starting from the google home screen and, since starting FROM a page implies navigating to it in the first place, this is a page altering operation which gets forwarded along to your browser to be invoked.

search_input <| textarea[name="q"] |
google <| https://google.com |
query_to_try <| "What is LSD.so?" |
search_button <| form center:nth-child(2) input[value="Google Search"] |
search_item_container <| div#search a |
search_item_link <| div#search a@href |

FROM google
|> ENTER INTO search_input query_to_try
|> CLICK ON search_button
|> GROUP BY search_item_container
|> SELECT query_to_try, search_item_link

The following two lines are where we operate the browser within the abstraction layer in which we are specifying the data to query for. Each expression follows the syntax of either {CLICK ON <selector>} or {ENTER INTO <selector> <text_of_interest>}

search_input <| textarea[name="q"] |
google <| https://google.com |
query_to_try <| "What is LSD.so?" |
search_button <| form center:nth-child(2) input[value="Google Search"] |
search_item_container <| div#search a |
search_item_link <| div#search a@href |

FROM google
|> ENTER INTO search_input query_to_try
|> CLICK ON search_button
|> GROUP BY search_item_container
|> SELECT query_to_try, search_item_link

The last two lines in this script are where we specify the fields we want to extract from the resulting website after the page altering operations; like what you would use in simpler LSD statements.

search_input <| textarea[name="q"] |
google <| https://google.com |
query_to_try <| "What is LSD.so?" |
search_button <| form center:nth-child(2) input[value="Google Search"] |
search_item_container <| div#search a |
search_item_link <| div#search a@href |

FROM google
|> ENTER INTO search_input query_to_try
|> CLICK ON search_button
|> GROUP BY search_item_container
|> SELECT query_to_try, search_item_link

Usage

To see a video of this in action, check out this demo we shared to Twitter where you will notice two things happening:

1. It is controlling the browser in the way described above
2. Rerunning the query at the end returns results quickly due to a caching strategy that is aware of page altering operations applied to a URL

Want to play around with it yourself? Here are the steps to take:

1. Open the browser after you have downloaded it locally
2. Press the key combination Command+Space to have the browser redirect you to our login page where you Google SSO in (or Control+Space for those on Linux)
3. You will be presented with a psql statement that contains connection credentials
4. Whether from the psql CLI in your terminal or by connecting to our database using a different approach, run the query you are interested in
5. You just tripped on LSD!

Usage from workbench

If instead of operating the browser as a developer, you can also go to our web workbench in your regular browser where, after logging in with Google SSO, you will be able to view/name past queries as well as run SQL statements with our credentials applied for you. In short, authenticating into our web workbench allows you to control the bicycle browser from the one you usually use.