Python data science 101, 05/01/2023, Alexander Dunkel

ad.vgiscience.org/lbsm-python-lecture/ <sup><a href="https://ad.vgiscience.org/lbsm-python-lecture/" class="smallsuper"><i data-customTooltip="The slides will remain online and after the lecture." class="fa fa-external-link"></i></sup></a>

Notes:
- Web presentation
- will remain online after the lecture

---

<p class="highlight">How to use the presentation:</p>
<i class="highlight fa fa-chevron-right"></i> Right arrow: Main Sections  
<i class="highlight fa fa-chevron-down"></i> Bottom arrow: Sub Slides  
<p> External links <a href="https://ad.vgiscience.org/lbsm-python-lecture/"> <sup class="smallsuper"><i data-customTooltip="Links to external further information, guides etc." class="fa fa-external-link"></i></sup></a></p>
<span class="hoverElement" data-customTooltip="This is a web presentation produced with Jekyll and reveal.js"> Hover on me to see tooltips </span>
<p>Press <kbd class="hoverElement" data-customTooltip="I mean your s-key, not this button.">S</kbd> to see slide notes.</p>

Notes:
- Broad topic, cannot be covered fully in one lecture
- overview; links to further information available

### Topics

<ol class="roman">
  <li>Data retrieval</li>
  <li>Data structures, formats, attribute mapping</li>
  <li>Types of incoming spatial data 
  <li>Data handling, aggregation</li>
  <li>Visualization</li>
  <li>Packages</li>
  <li>Environment setup and tools </li>
  <li>Code conventions</li>
  <li>Tracking: git, revision control</li>
  <li>Reproducibility: docker, WSL</li>
</ol>

Notes:
- structure of the first part (i.-v.) according to common steps
- second part introduction to background tools & concepts

### Workshops

Notes:
- announce upcoming workshops

<span class="highlight">**Jupyter Lab (Online/Hybrid) Workshop**</span>  
Social Media, Data Science, & Cartography

**07.30 - 09.00 am:**

- <span class="highlight">14./15.12.2021 (Tuesday/Wednesday)</span> 1st double period (DS)
- <i class="hoverElement" data-customTooltip="A few hours are needed until your resources are allocated. Please register ahead of time. The link will be renewed once in a while.">Please register yourself through the link (see email)</i>
- Have your TUD VPN connection ready

Notes:
- please register yourself ahead of time
- prepare VPN connection to TUD (OpenVPN, Cisco)

## Why Python?

- no vendor-lock in
- mature cartography & data science packages
- interoperability

Notes:
- Example: issues with vendor-lock in when switching workplaces (e.g. ESRI)
- compatibility, ease of integration into existing workflows

<div style="width:100%;position:fixed;bottom:100px;text-align:right;z-index:9999;">
<p style="vertical-align: bottom;" >
<strong class="highlight">The goal: </strong></br>
Structured </br>workflows </br>to process data.</br>
</p> 
</div>

Notes:
- Complex workflows: Not an issue, common for most cartographic visualization
- Overal Goal: Structured, reproducible processing pipelines

<div style="width:95%;position:fixed;bottom:100px;text-align:right;z-index:9999;">
<p style="vertical-align: bottom;" >
<strong>The reality</strong> <sup><a href="https://pythondatalab.wordpress.com/machine-learning/machine-learning-roadmap/" class="smallsuper"><i data-customTooltip="MACHINE LEARNING ROADMAP" class="fa fa-external-link"></i></a></sup>
</p> 
</div>

Notes:
- In reality, retrieval of data is a time consuming task
- 50% or more project time needed
- increasing when Social Media data is used
- not enough time for the more complicated, second part (data processing)

## Retrieval & Data filtering

Notes:
- following slides: overview of the critical tasks
- Explain common issues and pitfalls during data retrieval

APIs

- Flickr (flickr.photos.search)
- Instagram (GraphQL)
- Twitter (streaming API)

Notes:
- Web APIs comparable, with custom syntax each
- e.g. Twitter: Mostly streaming based (listening necessary)

Example: <span class="highlight">Instagram API</span>

```python
hashtag = "park"
query_url = f'https://www.instagram.com/explore/tags/{hashtag}/?__a=1'
```

<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">?__a=1</code>: return as json</li>
<li class="fragment" data-fragment-index="2"><code class="highlight">f'{var}'</code>: f-strings <sup><a href="https://docs.python.org/3/tutorial/inputoutput.html#formatted-string-literals" class="smallsuper"><i data-customTooltip="> Python 3.6: Formatted String Literals." class="fa fa-external-link"></i></sup></a></li>
<li class="fragment" data-fragment-index="3">Live Query to the API <sup><a href="https://www.instagram.com/explore/tags/park/?%5f%5fa=1" class="smallsuper"><i data-customTooltip="Works with & without login." class="fa fa-external-link"></i></sup></a></li>
</ul>

Notes:
- Instagram API sample
- Web API, GraphQL, discontinued but still publiy (why?)
- Consumer views endpoint as a url (path)
- API processes requests based on the path/endpoint

```python
import json, requests

response = requests.get(url=query_url)

if not response.status_code == 429:
    json_text = response.text
    json_data = json.loads(json_text)
```

<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">requests</code> <sup><a href="https://docs.python-requests.org/en/latest/" class="smallsuper"><i data-customTooltip=">Built for human beings." class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="2"><code class="highlight">json</code> Module <sup><a href="https://docs.python.org/3/library/json.html#module-json/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="3">HTML <code class="highlight">Status Codes</code> <sup><a href="https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="4">Frequent codes: 400, 403, 429, 500, 503, 504</li>
</ul>

Note:
- python can be used (requests library),
- similar to how the browser interacts with the API
- HTML error codes: 400 Bad requests, 403 Forbidden, 429 Too Many Requests, 500 Internal Server Error, 503 Service Unavailable, 504 Gateway Timeout

```python
json.dumps(json_data, indent=2)[0:550]
```

json: <span class="highlight">Flexible, structured, nested</span> data format
```json
{
  "graphql": {
    "hashtag": {
      "id": "17841563668080313",
      "name": "park",
      "allow_following": false,
      "is_following": false,
      "is_top_media_only": false,
      "profile_pic_url": "https://scontent-frt3-1.cdninstagram.com/v/t51.2885-15/e15/c179.0.721.721a/s150x150/166744655_1719090121604264_9033828806473210135_n.jpg?tp=1&_nc_ht=scontent-frt3-1.cdninstagram.com&_nc_cat=108&_nc_ohc=EBg1uEzip80AX_Za9Ag&ccb=7-4&oh=5d812109d80baaa71a671236d9daaa3d&oe=608E3E1B&_nc_sid=4efc9f",
      "edge_hashtag_to_media": {
        "cou ...
```

<div class="fragment" data-fragment-index="1" style="width:95%;position:fixed;bottom:10px;text-align:right;z-index:9999;">
<p style="vertical-align: bottom;" >
Open structure means not very suited for visualization.</p> 
</div>

Notes:
- common return format: json
- attribute–value pairs <code>"attribute":"value"</code>
- array (list) `[{object},{object}]`
- nesting makes it flexible, but problematic for visualization (missing standardization)

JSON <i class="highlight fa fa-arrow-right"></i> CSV

Notes:
- common raw interchange format for visualizations: CSV

<div style="width:95%;position:fixed;bottom:100px;text-align:right;z-index:9999;">
<p style="vertical-align: bottom;" >
<strong>Flickr CC </br>100 Million </br>Dataset (CSV) </strong> <sup><a href="https://lbsn.vgiscience.org/tutorial/yfcc-introduction/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>
</p> 
</div>

Notes:
- Social Media CSV example: Flickr YFCC 100 Million dataset
- published in 2014 by Yahoo/Flickr, based on CC By images
- same data can be retrieved using the public Flickr API

Example: Flickr CC100M CSV

```python
from pathlib import Path

root = Path.cwd()
csv = root / 'data' / 'yfcc100m.csv'
```
<ul>
<li>Define path to CSV</li>
<li class="fragment" data-fragment-index="1"><code class="highlight">pathlib.Path</code>: OS independent filesystem paths <sup><a href="https://docs.python.org/3/library/pathlib.html#basic-use" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="2">Size of dataset: <code class="highlight">~60 GB</code></li>
</ul>

Notes:
- basic code to open the CSV file
- use of pathlib!
- Is it a good idea to open a 60GB file at once?

Load CSV
```python
import pandas as pd

df = pd.read_csv(csv, encoding='utf-8', sep='\t', header=None, nrows=5)
```

<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">Pandas</code> - Python's Excel (understatement) <sup><a href="https://pandas.pydata.org/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="2"><code class="highlight">df</code>? DataFrame? 10 minutes to pandas <sup><a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="3">Docs: <code class="highlight">pd.read_csv()</code> <sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Notes:
- pandas to the rescue: Importance of DataFrame in current data science
- streaming of CSV file (chunked processing)
- explain basic syntax

Preview
```python
df.head()
```

Notes:
- Easy preview
- immediate observations: Missing header
- Public dataset, information difficult to find about meaning of columns
- common situation; other issues observable

Standardization: Attribute Mapping

Notes:  
- conversion JSON to tabular data structure makes standardization necessary
- not a complicated but time intensive task

<div style="width:95%;position:fixed;bottom:50px;text-align:right;z-index:9999;">
<p class="highlight" style="vertical-align: bottom;" data-customTooltip="A common language independent, privacy-aware and cross-network social-media data scheme." >
<strong>lbsn.vgiscience.org</strong> <sup><a href="https://lbsn.vgiscience.org/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>
</p> 
</div>

Notes:
- Integration of data
- time intensive, low-level task
- ability to create visualizations independent of data source

YFCC100M: First item.
```python
df.loc[0]
```
<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">loc-Indexer</code>: Access rows and columns by label(s) or a boolean array. <sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.loc.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

<div class="fragment" data-fragment-index="5" style="width:100%;position:fixed;top:50px;right:0px;text-align:right;z-index:9999;">
<p class="highlight" style="vertical-align: bottom;">
<strong>Flickr </br>API </br>Docs</strong></br> <sup><a href="https://www.flickr.com/services/api/flickr.photos.geo.photosForLocation.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>
</p> 
</div>

Notes:
- explain DataFrame index
- importance of loc indexer compared to other selection methods
- geospatial data peculiarities in Flickr

<span class="highlight">Other methods for deriving spatial information:</span>

<ul>
<li class="fragment" data-fragment-index="1">Public "Home Location" on Flickr user profiles: </br>Text-based description of a location (<span class="highlight" data-customTooltip=">Example Flickr user Profile (me) - with false information (!)">Geocoding <sup><a href="https://www.flickr.com/people/64974314@N08/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span>)</li>
<li class="fragment" data-fragment-index="2">Free-text descriptions of places (e.g. in posts) (<span class="highlight">Geoparsing <sup><a href="https://www.southampton.ac.uk/~sem03/SIGIR-2019-poster-v2.pdf" class="smallsuper"><i data-customTooltip=">Overview by Stuart et al. (2019)" class="fa fa-external-link"></i></a></sup></span>)</li>
<li class="fragment" data-fragment-index="3">Language of a post (<span class="highlight">Language detection</span>)</li>
<li class="fragment" data-fragment-index="4">Based on past user behavior (e.g. <span class="highlight">Bayesian Inference <sup><a href="https://www.researchgate.net/publication/326192551_Location_Inference_for_Non-Geotagged_Tweets_in_User_Timelines" class="smallsuper"><i data-customTooltip=">Li et al. (2018)" class="fa fa-external-link"></i></a></sup></span>)</li>
<li class="fragment" data-fragment-index="5"><span class="highlight">Gazetteer</span>: A database of places</li>
<li class="fragment" data-fragment-index="5">...</li>
</ul>

Notes:
- overview for other approaches to derive spatial information
- hint towards issues (ambiguity, privacy, complexity)

Gazetteer example: <span class="highlight">Instagram API</span>

```python
location = "1893214"
query_url = f'https://www.instagram.com/explore/locations/{location}/?__a=1'
```

<ul>
<li>Link to query <sup><a href="https://www.instagram.com/explore/locations/1893214/?%5f%5fa=1" class="smallsuper"><i class="fa fa-external-link"></i></sup></a></li>
</ul>

<pre><code>
{"graphql":null,"native_location_data":{"location_info":{"location_id":"1893214","facebook_places_id":"119533068103485","name":"Gro\u00dfer 
Garten","phone":"03 51-4456-600","website":"http://www.grosser-garten-dresden.de/","category":"Public 
Square / Plaza","price_range":2,"hours":{"status":"Open 24 hours","current_status":"Open 
24 hours","hours_today":"","schedule":[]},"lat":51.037697,"lng":13.762865,"location_address":"Hauptallee 
10","location_city":"Dresden, Germany","location_region":0,"location_zip":"01219","ig_business":{},"show_location_page_survey":false,"num_guides":null,"has_menu":false,"page_effect_info":{"num_effects":0,"thumbnail_url":null,"effect":null}},"ranked":{"sections":[{"layout_type":"media_grid","layout_content":{"medias":[{"media":{"taken_at":1635240067,"pk":2692911326172298695,"id":"2692911326172298695_1677335629","device_timestamp":163524001910942,"media_type":1,"code":"CVfJJVgID3H","client_cache_key":"MjY5MjkxMTMyNjE3MjI5ODY5NQ==.2","filter_type":0,"is_unified_video":false,"location":{"pk":1893214,"short_name":"Gro\u00dfer 
Garten","facebook_places_id":119533068103485,"external_source":"facebook_places","name":"Gro\u00dfer 
Garten","address":"Hauptallee 10","city":"Dresden, Germany","has_viewer_saved":null,"lng":13.762865,"lat":51.037697,"is_eligible_for_guides":true},"lat":51.037697,"lng":13.762865,"user":{"pk":1677335629,"username":"radio_dresden","full_name":"Radio 
[...]
</code></pre>

</div>

Notes:
- location endpoint
- coupling between Instagram and Facebook Places database
- open in Firefox tab and explain json structure

Notes:
- broad overview, more can be done
- amount of work depends on specific context and scrutiny of the analyst

<ol class="roman">
  <li class="highlight">Data retrieval</li>
  <li class="highlight">Data structures, formats, attribute mapping</li>
  <li class="highlight">Types of incoming spatial data 
  <li>Data handling, aggregation</li>
  <li>Visualization</li>
  <li>Packages</li>
  <li>Environment setup and tools </li>
  <li>Code conventions</li>
  <li>Tracking: git, revision control</li>
  <li>Reproducibility: docker, WSL</li>
</ol>

<div style="width:100%;position:fixed;top:200px;right:0px;text-align:right;z-index:9999;">
<p class="highlight" style="vertical-align: bottom;">
<strong>First part</strong><sup><i class="fa fa-check"></i></sup>
</p> 
</div>

Notes:
- brief break; ask questions
- Why is the Instagram API still publicly accessible, without authentication?

<ol class="roman">
  <li>Data retrieval</li>
  <li>Data structures, formats, attribute mapping</li>
  <li>Types of incoming spatial data 
  <li class="highlight">Data handling, aggregation</li>
  <li class="highlight">Visualization</li>
  <li>Packages</li>
  <li>Environment setup and tools </li>
  <li>Code conventions</li>
  <li>Tracking: git, revision control</li>
  <li>Reproducibility: docker, WSL</li>
</ol>

Aggregation Example: Usercount per Country

Notes:
- introduce simple application example:
- summarize user counts per country (YFCC dataset)

Two high-level tasks

1. Assign 100x100 km cells to countries <span class="fragment" data-fragment-index="1">(<span class="highlight">A geometric operation</span>)</span>
2. Count distinct users per country <span class="fragment" data-fragment-index="1">(<span class="highlight">A summary/aggregation operation</span>)</span>

Notes:
- complex task: divide into subtasks

Supplementary data <i class="highlight fragment fa fa-arrow-right" data-fragment-index="1"> Integration of information</i>

Notes:
- often, integration of data necessary, 
- based on additional, external information

Define origin of supplementary data in your code:
```python
DATA_PATH = Path.cwd() / "data"
FILENAME = "ne_50m_admin_0_map_subunits.shp"
```
<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">Natural Earth</code> map units shapefile (1:50m) <sup><a href="https://www.naturalearthdata.com/downloads/50m-cultural-vectors/50m-admin-0-details/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
</ul>

Notes:
- reproducibility also means that origin of data is tracked
- e.g. update/new release

Preview data
```python
import geopandas as gp
world = gp.read_file(
    DATA_PATH / FILENAME)
world.plot()
```

<ul>
<li><code class="highlight">Geopandas</code>: DataFrame extension for geospatial data<sup><a href="https://geopandas.org/en/stable/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment showmap" data-fragment-index="1"><code class="highlight">matplotlib.pyplot.plot</code><sup><a href="https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.plot.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Notes:
- fast preview generation of geospatial data
- with geopandas and matplotlib
- Explain geopandas as a spatial extension to pandas
- Access to all methods of pandas, spatial

Define <span class="highlight">full</span> origin of supplementary data in your code:
```python
DATA_PATH = Path.cwd() / "data"
URI = \
    "https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/50m/cultural/"
FILENAME = "ne_50m_admin_0_map_subunits.zip"
```

Notes:
- Improvement: define _full_ origin of data

This includes unzipping!

<span class="fragment" data-fragment-index="1">Unzipping? In python? Without writing to disk?</span><span class="fragment highlight" data-fragment-index="2"> <img class="plain highlight" style="width:75px;bottom:-10px" src="assets/grin-stars-solid.svg"/> Google <sup><a href="https://www.google.com/search?q=python+unzipping+a+.zip+file+without+writing+to+disk" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span>

Notes:
- ReadTheDocs or ask Google?
- A single Package: ReadTheDocs
- Broader questions (packages unknown): Google is Ok, but..

Notes:
- Python is 20 years old
- all kinds of knowledge from different versions,
- people with different skills
- makes it very difficult and problematic to copy&paste

Four lines of code:
```python
import requests, zipfile, io
r = requests.get(f'{URI}{FILENAME}')
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall(DATA_PATH)
```

Notes:
- explore options, but consciously write code

<span class="fragment fade-out" data-fragment-index="1">Counting users</span>

<div class="highlight" style="width:100%;position:fixed;bottom:50px;left:50px;text-align:left;z-index:9999;background-color: rgba(255,255,255, 0.8);">
<ul style="vertical-align: bottom;">
<li class="fragment" data-fragment-index="1">Background: Pre-aggregated grid with a set of user for each cell</li>
<li class="fragment" data-fragment-index="2">Foreground: Country shapes</li>
</ul>
</div>

Notes:
- back to application context
- illustrate starting situation

```python
users_set = set()
user = "96117893@N05"
users_set.add(user)
print(len(users_set))
> 1
```
Counting distinct users: Low-level approach

Notes:
- common approach to divide complex tasks into simple sub-tasks
- simplest way to count distinct users in python

Intersection

```python
grid_overlay = gp.overlay(
    grid, world, 
    how='intersection')
```

<ul>
<li><code class="highlight">grid</code> & <code class="highlight">world</code>: GeoDataFrames<sup><a href="https://geopandas.org/en/stable/docs/reference/geodataframe.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
<li class="fragment" data-fragment-index="1"><code class="highlight">Overlay operations</code>: </br>‘intersection’, ‘union’, ‘identity’, ‘symmetric_difference’ or ‘difference’ <sup><a href="https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.overlay.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
</ul>

Notes:
- common geometric operations, similar to standard GIS
- explain GeoDataFrame

Preview Intersection
```python
country_col = 'su_a3' # Column with country codes
query = grid_overlay[country_col].isin(["DEU", "FXX"])
sel = grid_overlay[query]
sel.plot(
    edgecolor='white',
    column=country_col,
    linewidth=0.3)
```
<ul>
<li><code class="highlight">query</code>: Boolean Indexing ("list" of True/False)<sup><a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#basics" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
<li class="fragment" data-fragment-index="1"><code class="highlight">df[query]</code>: Create a "slice" of the DataFrame</span></li>
</ul>

Notes:
- explain chaining of operations
- preview generation
- possible to split complex tasks further into separate lines

```python
grid.plot(edgecolor='white', column=country_col, figsize=(22,28), linewidth=0.3)
```

Notes:
- final result of the intersection

Count users:
```python
usercounts_country = grid["user_guid"].groupby(
    country_col).size()
```
<ul>
<li><code class="highlight">grid["user_guid"]</code> A single column (<code class="highlight">pd.Series</code>)<sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="1"><code class="highlight">pd.Series.groupby()</code> operation<sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.groupby.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="2"><code class="highlight">size()</code> Return the number of elements in the underlying data.<sup><a href="https://pandas.pydata.org/docs/reference/api/pandas.Series.size.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Notes:
- Remaining step: Counting/aggregating users per country
- explain pandas operations

Count distinct users:
```python
series_grouped = grid["user_guid"].groupby(
        country_col).apply(set)
usercounts_country = series_grouped.apply(len)
world.loc[
    usercounts_country.index,
    "usercount_est"] = usercounts_country
```
<ul>
<li class="fragment" data-fragment-index="1"><code class="highlight">grid["user_guid"]</code> 1. Take column "user_guid" </li>
<li class="fragment" data-fragment-index="2"><code class="highlight">.groupby(country_col)</code> 2. Group by "country" column </li>
<li class="fragment" data-fragment-index="3"><code class="highlight">.apply(set)</code> 3. Apply "set" to resulting value-groups </li>
<li class="fragment" data-fragment-index="4"><code class="highlight">.apply(len)</code> 4. Length of sets (number of users) </li>
</ul>

Notes:
- Users may be present multiple times
- count distinct: similar operations
- recapitulate simple low-level user counting with sets

Visualize data

</br>
```python
import geoviews as gv
gv.Polygons(
    world, crs=crs.Mollweide(), vdims='usercount_est')
```

<ul class="fragment" data-fragment-index="1">
<li class="highlight" style="background-color: rgba(255,255,255, 0.9);">Geoviews<sup><a href="https://geoviews.org/index.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Notes: 
- fast preview with matplotlib during development
- sharing with others: Interactive Visualizations
- Geoviews compatible with GeoDataFrames (ad-hoc)

</br>
</br>
</br>
</br>
</br>
```python
gv.Polygons(world, crs=crs.Mollweide(), vdims='usercount_est') \
    +  gv.Path(world, crs=crs.Mollweide())
```
<ul class="fragment" data-fragment-index="1">
<li>Stack or combine plots<sup><a href="https://geoviews.org/user_guide/Geometries.html#feature" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Notes:
- explain basic syntax for map layout/layers in gv

Export to HTML file
```python
import holoviews as hv
hv.save(layers, 'sample.html', backend='bokeh')
```

<div class="fragment highlight" data-fragment-index="1" style="width:100%;position:fixed;bottom:50px;left:0px;text-align:left;z-index:9999;">
<p>Workshop </br>result</p>
</div>

Notes:
- Pyviz: Visualization packages with compatible APIs
- Holoviews: high-level plotting API to Bokeh interactive maps
- example: Generate standalone archive HTML from Geoviews Layers

<ul>
  <li ><span class="highlight">Export</span> data to shapefiles <span class="highlight">at any time</span></li>
  <li >& continue layouting in <span class="highlight">QGis</span> (or ArcMap etc.)</li>
  <li class="fragment" data-fragment-index="1"><span class="highlight">Integrate python gradually</span>, by replacing small parts of your workflow</li>
</ul>

Notes:
- Unlike proprietary software, interoperability is a strength
- Incremental and gradual integration of python into existing workflows

<ol class="roman">
  <li >Data retrieval</li>
  <li >Data structures, formats, attribute mapping</li>
  <li >Types of incoming spatial data 
  <li class="highlight">Data handling, aggregation</li>
  <li class="highlight">Visualization</li>
  <li>Packages</li>
  <li>Environment setup and tools </li>
  <li>Code conventions</li>
  <li>Tracking: git, revision control</li>
  <li>Reproducibility: docker, WSL</li>
</ol>

<div style="width:100%;position:fixed;top:256px;right:0px;text-align:right;z-index:9999;">
<p class="highlight" style="vertical-align: bottom;">
<strong>Second part</strong><sup><i class="fa fa-check"></i></sup>
</p> 
</div>

Notes:
- Further steps not shown here is extensibility through extensions 
- & customizations of visualization (open ended topic)

<ol class="roman">
  <li>Data retrieval</li>
  <li>Data structures, formats, attribute mapping</li>
  <li>Types of incoming spatial data 
  <li>Data handling, aggregation</li>
  <li>Visualization</li>
  <li class="highlight">Packages</li>
  <li class="highlight">Environment setup and tools </li>
  <li class="highlight">Code conventions</li>
  <li class="highlight">Tracking: git, revision control</li>
  <li class="highlight">Docker, WSL</li>
</ol>

<div style="width:100%;position:fixed;bottom:150px;text-align:right;z-index:9999;">
    <p class="fragment" data-fragment-index="1" style="vertical-align: bottom;" >
        <span class="highlight"><i class="highlight fa fa-arrow-right"></i></span> Chore, </br>Orchestration
    </p>
    <p class="fragment" data-fragment-index="2" style="vertical-align: bottom;" >
        <span class="highlight"><i class="highlight fa fa-arrow-right"></i></span> Main goal: </br><u>Reproducibility</u>
    </p>
</div>

Notes:
- commonly not covered at university
- frequent changes; largely depends on application context
- singular, opinionated perspective

<div style="width:100%;position:fixed;bottom:150px;text-align:right;z-index:9999;">
    <p class="fragment" data-fragment-index="1" style="vertical-align: bottom;" >
        <span class="highlight">Save your </br>friends from </br>dependency hell </br><sup><a href="https://towardsdatascience.com/devops-for-data-science-making-your-python-project-reproducible-f55646e110fa" class="smallsuper"><i data-customTooltip=">Devops for Data Science: Making your Python Project Reproducible" class="fa fa-external-link"></i></a></sup></span>
    </p>
</div>

Notes:
- this is where all python developers have been at some point

<div style="width:100%;position:fixed;bottom:25px;text-align:center;z-index:9999;" class="fragment" data-fragment-index="1">
<p style="vertical-align: bottom;background-color:white" >
<span class="highlight">An opinionated list of packages for cartography <sup><a href="https://gitlab.vgiscience.de/lbsn/tools/jupyterlab" class="smallsuper"><i data-customTooltip=">IfK Jupyter Lab Container" class="fa fa-external-link"></i></a></sup></span></p> 
</div>

Notes:
- starts with choice of packages (carto example)
- recommendation: Choose a set of packages that are compatible
- particularly: where developers care for interoperability

Environments and package managers (2021)

<ul>
<li><span class="fragment strike" data-fragment-index="1">Anaconda</span> <span class="highlight fragment" data-fragment-index="1"><span>Miniconda</span> for data science <sup><a href="https://docs.conda.io/en/latest/miniconda.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
<li class="fragment" data-fragment-index="2"><span class="highlight">Poetry</span> & pyproject.toml <sup><a href="https://python-poetry.org/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Note:
- highly controversial topic
- 2021: Conda still _Ok_ for data science
- most promising for future and python package managing overall: Poetry

Sample <span class="highlight">environment.yml (conda)</span>
```yml
name: sampleenv
channels:
  - conda-forge
  - defaults
dependencies:
  - python>=3.7
  - pip
  - colorcet
  - 'geoviews-core=1.8.1'
  - datashader
  - geopandas
  - seaborn
  - xarray
  - pip:
    - python-hll
```

Notes:
- Even for conda, many ways exist to handle packages
- Example: Custom conda `.yml` format
- describe syntax

Restore environment

Notes:
- Process for restoring a conda environment from a environment.yml

Code conventions (PEP)

Notes:
- Mistake of ignoring code conventions as a beginner
- Motivations for using code conventions

<ul>
<li>Following conventions improves the readability of code</li>
<li class="fragment" data-fragment-index="1">PEP 8 -- Style Guide for Python Code <sup><a href="https://www.python.org/dev/peps/pep-0008/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="2">Use <span class="hoverElement" data-customTooltip="Lint, or a linter, is a static code analysis tool used to flag programming errors, bugs, stylistic errors (etc.).">linters</span>: Pylint <sup><a href="https://github.com/PyCQA/pylint" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>, pycodestyle <sup><a href="https://github.com/PyCQA/pycodestyle" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>, black <sup><a href="https://github.com/psf/black" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>
</ul>

Notes:
- not necessary to follow all conventions always
- be conscious

Reproducibility: Comments

```python
import requests, zipfile, io
r = requests.get(f'{URI}{FILENAME}')
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall(DATA_PATH)
```

Notes:
- Example procedure from earlier

```python
import requests, zipfile, io
# Get zip file and extract
r = requests.get(f'{URI}{FILENAME}')
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall(DATA_PATH)
```

Notes:
- Add a comment: Increase reproducibility

```python
import requests, zipfile, io

def get_zip_extract(
    uri: str, filename: str, data_path: Path):
    """Get Zip file and extract to data_path."""
    r = requests.get(f'{uri}{filename}')
    z = zipfile.ZipFile(io.BytesIO(r.content))
    z.extractall(data_path)

get_zip_extract(URI, FILENAME, DATA_PATH)
```

<ul>
<li class="fragment" data-fragment-index="1">PEP 257 -- Docstring Conventions <sup><a href="https://www.python.org/dev/peps/pep-0257/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
<li class="fragment" data-fragment-index="2"><code class="highlight">uri: str</code> Python type hints</span><sup><a href="https://docs.python.org/3/library/typing.html" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Notes:
- improvement: Separate "business" logic from actual operation
- reusability of code: reduce duplication
- docstrings, type-hints: Powerful feature,
- used e.g. to in automatically generate APIs and documentation (Fastapi)

Tools:
<ul>
<li class="fragment" data-fragment-index="1"><span class="fragment strike" data-fragment-index="2">Visual Studio</span>, <span class="fragment fade-in strike" data-fragment-index="3">PyCharm</span><span class="fragment" data-fragment-index="4">, </br><span class="highlight">VSCode</span> - Standalone code development <sup><a href="https://code.visualstudio.com/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></span></li>
<li class="fragment" data-fragment-index="5"><span class="highlight">Jupyter Lab </span> - Combine comments and code <sup><a href="https://jupyter.org/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></li>
</ul>

Notes:
- 2021 summary of the most important tools for coding

<div style="width:100%;position:fixed;bottom:50px;text-align:center;z-index:9999;">
<p style="vertical-align: bottom;background-color:white" >
VSCode started from WSL</p> 
</div>

Notes:
- Common convention for Windows developers: Frontend through Web/Windows, Backend: Linux
- Why Linux? Reproducibility.
- Example: VSCode started in a folder through WSL/Linux

<div style="width:100%;position:fixed;bottom:50px;text-align:center;z-index:9999;">
<p style="vertical-align: bottom;background-color:white" >
Jupyter Lab (Workshop)</p> 
</div>

Notes:
- Jupyter recommended for beginners

Notes:
- combine code & comments; process oriented workflow
- suited for gradual code and visualization development

Faster Development of Visualizations

Notes:
- Example changing color of grid
- immediately see visual effect of changes to code

IPython makes it easier to learn

Notes:
- example of anti-pattern to use print() for variable inspection
- in jupyter, use of `?variable` for inspection
- IPython: Runtime environment oriented

Parametrized Visualizations

<div style="width:100%;position:fixed;bottom:50px;text-align:center;z-index:9999;" class="fragment" data-fragment-index="1">
<p style="vertical-align: bottom;background-color:white" >
(Google) Best practices: From Jupyter notebooks to production ready code<sup><a href="https://cloud.google.com/blog/products/ai-machine-learning/best-practices-that-can-improve-the-life-of-any-developer-using-jupyter-notebooks" class="smallsuper"><i class="fa fa-external-link"></i></a></sup></p> 
</div>

Notes:
- Conventions for writing notebooks (Example: Google best Practice)
- Example: Parameterized Notebooks (50km to 500km grid)
- specify parameters, to be changed by users, at the beginning
- prepare code accordingly

Direct Export to HTML to share results

Notes:
- Interoperability
- Archive & Sharing of Code, Comments & Output in a single format

<span class="highlight">Benefits in summary:</span>
- Reproducibility of the analytical process
- Reusability of code
- Code that is commented and directly connected to generated visuals
- Remote execution of code, local development in browser

<span class="hoverElement" data-customTooltip="Lightweight, standalone container as an executable package of software that includes everything needed to run an application">Docker</span>, <span class="hoverElement" data-customTooltip="Windows Subsystem for Linux">WSL</span> <sup><a href="https://ad.vgiscience.org/links/posts/2019-05-27-setup-wsl-win10/" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>, <span class="hoverElement" data-customTooltip="Code revision tracking">Git</span> <sup><a href="https://ad.vgiscience.org/links/posts/2019-05-27-wsl-software/#git" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>?  
<img class="plain highlight" style="width:75px;bottom:-10px;opacity: 0.5;" src="assets/grimace-regular.svg"/>

Notes:
- there is more
- not necessary to start with everything;
- take small steps, integrate gradually

<span class="highlight">IfK Jupyter Lab Container</span> <sup><a href="https://gitlab.vgiscience.de/lbsn/tools/jupyterlab" class="smallsuper"><i class="fa fa-external-link"></i></a></sup> & <span class="highlight">Workshop Notebooks</span> <sup><a href="https://gitlab.vgiscience.de/ad/mobile_cart_workshop2020" class="smallsuper"><i class="fa fa-external-link"></i></a>  </sup>
- Environment prepared with conda
- Notebooks accessed through web browser
- Each notebook is started on a separate machine (ZIH)
- Notebook revisions tracked in git

Notes:
- For cartographers: Have a look at our Jupyter Workflow
- Can be started at home

See instructions to run the workshop <span class="highlight">at home</span> <sup><a href="https://gitlab.vgiscience.de/ad/mobile_cart_workshop2020/-/blob/master/resources/faq/04_offline_setup.md" class="smallsuper"><i class="fa fa-external-link"></i></a></sup>