Analyzing Your Browser History Using Python and Pandas (applecrazy.github.io)

Vivaldi does this for me in the history page itself. Pretty weird to use Python just to see a distribution of what sites you visited.

Iirc stock chrome also has a way to view this.

> Since Pandas probably doesn’t do this out of the box

It does, use df[column_name].str.split(expand=True), where df is your DataFrame.

Great post for a quick read! You could maybe have taken your google searches from within the URL and Stack Overflow question titles and done word frequency analysis and maybe learned what languages you code in most often/what type of problems you have most as well. Good intro for a quick read though.

It seems a bit odd to export the data in the command line to then work with it in pandas, given that it's a stock sqlite3 database. I'd just use sqlite3.connect and pd.read_sql_query directly. (I know, the db will be locked when Chrome is running, but you can always shutil.copy it to a temp location.)

I'm learning Python currently and this is the sorta stuff that's neat to see!

I haven't had any reason to use pandas yet so it'll be a good excuse.

It's a project that's not very long, requires data everyone has and gives a cool insight at the end.

Nice!

I think the location on Linux (Ubuntu 16.04) is:

    ~/.config/google-chrome/Default

or:

    ~/.config/chromium/Default

Neat. As a Firefox user, it appears that this does the trick:

    sqlite3 ~/.mozilla/firefox/$YOUR_PROFILE_ID/places.sqlite "SELECT datetime(visit_date/1000000,'unixepoch'),url FROM moz_historyvisits JOIN moz_places ON place_id=moz_places.id ORDER BY visit_date DESC" > hist.txt

(Looks like the timestamp is in UTC here.)

----

Does the original post's code perhaps count two visits to the same URL as one? Something like

    SELECT datetime(visit_time/1000000-11644473600,'unixepoch'),urls.url 
    FROM visits JOIN urls ON visits.url=urls.id
    ORDER BY visit_time DESC;

might be worth a try.

These two loops:

  raw_data = [line.split('|', 1) for line in [x.strip() for x in content]]

can be simplified to a single loop:

  raw_data = [line.strip().split('|', 1) for line in content]

Using str.replace here is also non-idiomatic:

  plt.title('Top $n Sites Visited'.replace('$n', str(topN)))

How about using str.format instead:

  plt.title('Top {n} Sites Visited'.format(n=topN))

  f"Top {topN} Sites Visited"

Thanks a lot for this code! If someone is interested and does not want to copy the code by themself, I copied the code into a jupyter notebook: https://github.com/davidgengenbach/chrome_history

I don't think that the pulled data is *extremely messy(. You can also read `hist.txt` with:

  pd.read_csv('hist.txt', sep='|', header=None)

You can skip the entire SQLite output to file by reading directly from the database: http://pandas.pydata.org/pandas-docs/version/0.20/generated/...

There's a video (now pay-walled) from Cameron Davidson Pilon of Bayesian Methods for Hackers fame, where he takes this approach to build a basic Markov Chain model to predict patterns in his browsing.

https://dataorigami.net/collections/all/products/create-mark...

This type of history stats is buildin out of the box in Vivaldi browser.

What about the cmd+shift+N stuff ;)

Just tried this, but am getting an error with the datetime. How could I fix this?

  Traceback (most recent call last):
    File "hist.py", line 14, in <module>
      data.datetime = pd.to_datetime(data.datetime)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 373, in to_datetime
      values = _convert_listlike(arg._values, True, format)
    File "/usr/local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py", line 306, in _convert_listlike
      raise e
  pandas._libs.tslib.OutOfBoundsDatetime: Out of bounds 
  nanosecond timestamp: 1601-01-01 00:00:00

The time on my Mac is correct, and datetime gets the correct time?

  data.loc[data.datetime == '1601-01-01 00:00:00', 'datetime'] = '1970-01-01 00:00:00'

I have a far more mundane use case - you have friends over, play songs off of youtube all night, after the fact, would like to publish a playlist on youtube, but it's not simple using browser history. I always figured I'd have to write a plugin and find clusters of short youtube sessions and go from there, but as a backend guy, I might play around with this stack instead.

Awesome first content post, man! I like yours better than mine: https://andythemoron.com/blog/2017-03-12/Scraping-For-Fun-An...

One note you may want to add (or step you may want to tweak) is that the initial sqlite3 command won’t work if chrome is currently open. I just copied the History folder to somewhere else and did the same thing, as per the superuser thread you helpfully linked to.