Moving rocks around

A short DLT tutorial

2026-01-18T15:30:00-03:00

Last year I stumbled upon dlt, a data loading tool that attempts to simplify the process of getting data without having to write too much code or spin up complex infrastructure.

I used it to fetch data from some sources at work, mostly from SOAP APIs and it is working quite well. I struggled to make sense of the docs at first and when I managed to get something working I was already so invested that I decided to write this short tutorial to help others get started.

I presented this at the PyCon Chile 2025 conference as well, and you can find the code and the tutorial contents in my GitHub repo at https://github.com/diegoquintanav/dlt-tutorial. I don't claim this to be the ultimate guide, but I hope it helps you get started.

The Pinochet-Rettig dataset, Revisited

2024-01-01T15:00:00+04:00

What

This is a follow up to the Pinochet-Rettig post. I've learned a lot since then and I wanted to revisit the dataset with a more robust approach. I was motivated also by the conmemoration of the 50th anniversary of the coup d'etat in Chile.

The project highlights and contributions as of now are:

dbt documentation
Elementary Report using Elementary
FastAPI Docs hosted on fly.io
Graphql endpoint using strawberry-graphql

I'm using dbt to transform the data and expose it as a REST API using FastAPI with a GraphQL layer, as well as a semantic layer for running SPARQL queries. I'm also serving the api using fly.io, which is a really cool service that is not charging me anything for this project (bills below USD 5 monthly are 100% free as of now).

The data is stored in a Postgres database with postgis, and everything is running in Docker containers.

Make sure to visit the Github repo and its README for more details.

I wish I had more time to work on this, but I'm happy with the results so far. Let me know if you want to contribute!

Python and Docker

2023-09-02T15:00:00+02:00

Update

Update

I have formatted this entry into a workshop format for Som Energia, the cooperative I work with at Girona. You can find it here.

What

I've toyed with docker a fair amount of time. There's a lot going on to get a working container so let me summarize a bit of what I'll show here:

We use a multistage docker build to separate the build (builder) stage from the final (app) stage and a development stage (dev) for development purposes.
We use pipx to install poetry and poetry to install our dependencies.
We pivot from the app-pre stage for doing development e.g. using devcontainers, and from the app stage for production.
We create a user and a group to avoid running as root at both levels. At both dev and prod levels, this user may or not may have access to sudo.

What follows are my dry, unpolished notes about how to do this. My goal is to turn this into a workshop at some point.

Docker image requirements

When building a docker image, we wish to achieve the following:

Security: we don't want to run as root in production. In development, we may or may not want to run as root. We achieve this by defining a user and a group and granting sudo access depending on the docker build stage. There are rootless options such as podman or even docker itself, but I haven't explored them yet.
Size: we want to keep the image size as small as possible. We achieve this by using a multistage build where the final image is based on a slim python image and it only contains the virtual environment with the dependencies, installed using poetry at a previous stage.
Speed: we want to keep the build time as small as possible. We can achieve this by leveraging docker caching and by separating the build stage from the final stage.
Flexibility: we want to be able to use the same image for development and production. We achieve this by defining a dev stage that is based on the app-pre stage and that allows us to install additional dependencies for development purposes.
Stability: we want to pin versions for stable builds. We achieve this by defining the versions of poetry and pipx that we want to use. We use a pyproject.toml file to define the versions of our dependencies.

I'm not going into docker tagging and versioning here, which is much more involved and I haven't figured out yet how to do it elegantly.

Example project

We'll use a simple python project with a pyproject.toml file and a poetry.lock file. We'll use poetry to install our dependencies.

mkdir -p myproject
cd myproject
touch pyproject.toml poetry.lock

Edit the pyproject.toml file to add some dependencies:

[tool.poetry]
authors = ["Your Name <your@email.ifyouwant>"]
description = "An example project to showcase docker and poetry"
name = "myproject"
version = "0.1.0"

[tool.poetry.dependencies]
python = ">=3.8,<3.11"
cowsay = "^4.0.0"

[build-system]
build-backend = "poetry.core.masonry.api"
requires = ["poetry-core"]

Global variables

We start by defining a global group of variables that will be used across many stages.

# ---------------------------------------------------------------------------- #
#                            global build arguments                            #
# ---------------------------------------------------------------------------- #

# Global ARG, available to all stages (if renewed)
ARG WORKDIR="/app"

# global username
ARG USERNAME=bluesmonk
ARG USER_UID=1000
ARG USER_GID=1000

# tag used in all images
ARG PYTHON_VERSION=3.8.9

We can reference these variables later at every stage by simply using the same ARG statement. For example, we can use ARG WORKDIR at every stage to reference the same value.

The `builder` stage

Give it a read first and then we'll go through it step by step.

# ---------------------------------------------------------------------------- #
#                                  build stage                                 #
# ---------------------------------------------------------------------------- #

FROM python:${PYTHON_VERSION}-slim AS builder

# Renew args
ARG WORKDIR
ARG USERNAME
ARG USER_UID
ARG USER_GID

# Poetry version
ARG POETRY_VERSION=1.5.1

# Pipx version
ARG PIPX_VERSION=1.2.0

# prepare the $PATH
ENV PATH=/opt/pipx/bin:${WORKDIR}/.venv/bin:$PATH \
 PIPX_BIN_DIR=/opt/pipx/bin \
 PIPX_HOME=/opt/pipx/home \
 PIPX_VERSION=$PIPX_VERSION \
 POETRY_VERSION=$POETRY_VERSION \
 PYTHONPATH=${WORKDIR} \
 # Don't buffer `stdout`
 PYTHONUNBUFFERED=1 \
 # Don't create `.pyc` files:
 PYTHONDONTWRITEBYTECODE=1 \
 # make poetry create a .venv folder in the project
 POETRY_VIRTUALENVS_IN_PROJECT=true

# ------------------------------ add user ----------------------------- #

RUN groupadd --gid $USER_GID "${USERNAME}" \
 && useradd --uid $USER_UID --gid $USER_GID -m "${USERNAME}"

# -------------------------- add python dependencies ------------------------- #

# Install Pipx using pip
RUN python -m pip install --no-cache-dir --upgrade pip pipx==${PIPX_VERSION}
RUN pipx ensurepath && pipx --version

# Install Poetry using pipx
RUN pipx install --force poetry==${POETRY_VERSION}

# ---------------------------- add code specifics ---------------------------- #

# Copy everything to the container
# we filter out what we don't need using .dockerignore
WORKDIR ${WORKDIR}

# make sure the user owns /app
RUN chown -R ${USER_UID}:${USER_GID} ${WORKDIR}

# Copy only the files needed for installing dependencies
COPY --chown=${USER_UID}:${USER_GID} pyproject.toml poetry.lock ${WORKDIR}/

USER ${USERNAME}

We start by defining the builder stage. We use a slim python image as a base image. We use the AS keyword to name the stage.
We renew the global variables that we defined earlier. We do this because we want to use the same variables at every stage. We can do this by simply using the same ARG statement.
We define some additional variables that we'll use later. We define the versions of poetry and pipx that we want to use. We do this because we want to pin versions for stable builds.
We define the $PATH variable to control and configure python, poetry and pipx installations beforehand
We configure pipx installation so that installer uses custom directories.
We also add the virtual environment directory to the path so that we can run the project from anywhere.
We define the PYTHONPATH variable to point to the project directory at /app so that we can import modules from the project.
We configure poetry using the POETRY_VIRTUALENVS_IN_PROJECT variable to make poetry create a .venv folder in the project, at /app/.venv.
We add a user and a group to avoid running as root. We don't use this user now, but we will later.
We install pipx using pip and later we install poetry using pipx.
We define our working directory at /app. This directory needs to be writable by the user that we created earlier. We also copy the pyproject.toml and poetry.lock files to the container.
We seal the stage by switching to the user that we created earlier. From this point on, all commands will be run as this user unless we switch to another user.

The `app-pre` stage

This stage starts from the builder stage and it installs only main dependencies using poetry. We pass the --no-root flag to poetry to avoid installing the project itself. We use this stage to install dependencies and to create a virtual environment at /app/.venv.

# ---------------------------------------------------------------------------- #
#                                 app-pre stage                                #
# ---------------------------------------------------------------------------- #

FROM builder AS app-pre

# Install dependencies and creates a a virtualenv at /app/.venv
RUN poetry install --no-root --only main

The `app` stage

This is mostly the same as the builder stage, but with a few changes:

We use a slim python image as a base image. We use the AS keyword to name the stage. At this point you can use any image you want, provided it has python installed. We make the image tag depend on the python version that we defined earlier.
We copy the directory at /app/.venv from the app-pre stage to the app stage. We do this because we want to keep the virtual environment that we created at the app-pre stage, but we don't need poetry and pipx anymore.
We don't need to install poetry and pipx anymore, so we remove them from the $PATH variable.

# ---------------------------------------------------------------------------- #
#                                   app stage                                  #
# ---------------------------------------------------------------------------- #

# We don't want to use alpine because porting from debian is challenging
# https://stackoverflow.com/a/67695490/5819113
FROM python:${PYTHON_VERSION}-slim AS app

# refresh global arguments
ARG WORKDIR
ARG USERNAME
ARG USER_UID
ARG USER_GID

# refresh PATH
ENV PATH=/opt/pipx/bin:${WORKDIR}/.venv/bin:$PATH \
 POETRY_VERSION=$POETRY_VERSION \
 PYTHONPATH=${WORKDIR} \
 # Don't buffer `stdout`
 PYTHONUNBUFFERED=1 \
 # Don't create `.pyc` files:
 PYTHONDONTWRITEBYTECODE=1

# ------------------------------ user management ----------------------------- #

RUN groupadd --gid $USER_GID "${USERNAME}" \
 && useradd --uid $USER_UID --gid $USER_GID -m "${USERNAME}"

# ------------------------------- app specific ------------------------------- #

WORKDIR ${WORKDIR}

RUN chown -R ${USER_UID}:${USER_GID} ${WORKDIR}

COPY --from=app-pre --chown=${USER_UID}:${USER_GID} ${WORKDIR} ${WORKDIR}

USER ${USERNAME}

ENTRYPOINT [ "python" ]
CMD [ "--version" ]

The `dev` stage

This stage starts from the app-pre stage and it installs all remaining dependencies using poetry. It looks like the app stage, but with a few changes:

We install development dependencies such as git, sudo, wget and iputils-ping. We do this because we may want to be able to install additional software after connecting to the container.
We install extra python dependencies for development purposes. We do this by adding a dev section to the pyproject.toml file and by passing the --no-root flag to poetry to avoid installing the project itself.
We add the user to the sudoers file. This is not recommended for production.
We seal the stage by switching to the user that we created earlier. This user has sudo access.

# ---------------------------------------------------------------------------- #
#                                   dev stage                                  #
# ---------------------------------------------------------------------------- #

FROM app-pre AS dev

# refresh global arguments
ARG WORKDIR
ARG USERNAME
ARG USER_UID
ARG USER_GID


USER root

# Add USERNAME to sudoers. Omit if you don't need to install software after connecting.
RUN apt-get update \
 && apt-get install -y sudo git iputils-ping wget \
 && echo ${USERNAME} ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/${USERNAME} \
 && chmod 0440 /etc/sudoers.d/${USERNAME}

USER ${USERNAME}

# install all remaning dependencies
RUN poetry install --no-root --with dev

USER ${USERNAME}

Buiding using docker compose

Edit a docker compose file to build the image at docker-compose.yml

---
name: docker-poetry-project

version: "3.7"
services:
  builder:
    image: "myproject-builder:latest"
    deploy:
      replicas: 0 # never start service, since it is only for build purposes
    build:
      context: .
      dockerfile: Dockerfile
      target: builder
      cache_from:
        - "myproject-builder:latest"
  app:
    image: "myproject-app:latest"
    deploy:
      replicas: 0 # never start service, since it is only for build purposes
    build:
      context: .
      dockerfile: Dockerfile
      target: app
      cache_from:
        - "myproject-builder:latest"
        - "myproject-app:latest"
  app-dev:
    image: "myproject-app-dev:latest"
    build:
      context: .
      dockerfile: Dockerfile
      target: dev
      cache_from:
        - "myproject-builder:latest"
        - "myproject-app-dev:latest"
    volumes:
      - python_venv_app_dev:/app/.venv # mount virtualenv at /venv. See https://stackoverflow.com/a/74015989/5819113
    command: /bin/sh -c "sleep infinity" # keep container running

volumes:
  python_venv_app_dev:

A few notes about this file:

Using the cache_from option, we can use the cache from previous builds. This is useful to speed up the build process.
Setting replicas: 0 ensures that the service is never started. We only use it for building the image.
We mount the virtual environment at /app/.venv using a named volume. This is very important because we don't want to share this directory with the host machine. This is useful for development purposes because otherwise you may run into permission issues when trying to install dependencies from the host machine (e.g. if you decide to use another user id or another group). See https://stackoverflow.com/a/74015989/5819113 for details.

Using this file, we can build the images using docker compose build.

Try out your new containers

Use docker compose to get a shell inside the app-dev container.

docker compose build

docker compose up -d app-dev

docker compose exec -it --entrypoint bash app-dev

python --version
python -c "import cowsay; cowsay.cow('Hello world')"
ls -la
exit

To get inside the app container, use docker compose run --rm -it --entrypoint bash app.

Full Dockerfile

# ---------------------------------------------------------------------------- #
#                      example usage for docker and poetry                     #
# ---------------------------------------------------------------------------- #


# ---------------------------------------------------------------------------- #
#                            global build arguments                            #
# ---------------------------------------------------------------------------- #

# Global ARG, available to all stages (if renewed)
ARG WORKDIR="/app"

# global username
ARG USERNAME=bluesmonk
ARG USER_UID=1000
ARG USER_GID=1000

# tag used in all images
ARG PYTHON_VERSION=3.8.9

# ---------------------------------------------------------------------------- #
#                                  build stage                                 #
# ---------------------------------------------------------------------------- #

FROM python:${PYTHON_VERSION}-slim AS builder

# Renew args
ARG WORKDIR
ARG USERNAME
ARG USER_UID
ARG USER_GID

# Poetry version
ARG POETRY_VERSION=1.5.1

# Pipx version
ARG PIPX_VERSION=1.2.0

# prepare the $PATH
ENV PATH=/opt/pipx/bin:${WORKDIR}/.venv/bin:$PATH \
 PIPX_BIN_DIR=/opt/pipx/bin \
 PIPX_HOME=/opt/pipx/home \
 PIPX_VERSION=$PIPX_VERSION \
 POETRY_VERSION=$POETRY_VERSION \
 PYTHONPATH=${WORKDIR} \
 # Don't buffer `stdout`
 PYTHONUNBUFFERED=1 \
 # Don't create `.pyc` files:
 PYTHONDONTWRITEBYTECODE=1 \
 # make poetry create a .venv folder in the project
 POETRY_VIRTUALENVS_IN_PROJECT=true

# ------------------------------ add user ----------------------------- #

RUN groupadd --gid $USER_GID "${USERNAME}" \
 && useradd --uid $USER_UID --gid $USER_GID -m "${USERNAME}"

# -------------------------- add python dependencies ------------------------- #

# Install Pipx using pip
RUN python -m pip install --no-cache-dir --upgrade pip pipx==${PIPX_VERSION}
RUN pipx ensurepath && pipx --version

# Install Poetry using pipx
RUN pipx install --force poetry==${POETRY_VERSION}

# ---------------------------- add code specifics ---------------------------- #

# Copy everything to the container
# we filter out what we don't need using .dockerignore
WORKDIR ${WORKDIR}

# make sure the user owns /app
RUN chown -R ${USER_UID}:${USER_GID} ${WORKDIR}

# Copy only the files needed for installing dependencies
COPY --chown=${USER_UID}:${USER_GID} pyproject.toml poetry.lock ${WORKDIR}/

USER ${USERNAME}


# ---------------------------------------------------------------------------- #
#                                 app-pre stage                                #
# ---------------------------------------------------------------------------- #

FROM builder AS app-pre

# Install dependencies and creates a a virtualenv at /app/.venv
RUN poetry install --no-root --only main

# ---------------------------------------------------------------------------- #
#                                   app stage                                  #
# ---------------------------------------------------------------------------- #

# We don't want to use alpine because porting from debian is challenging
# https://stackoverflow.com/a/67695490/5819113
FROM python:${PYTHON_VERSION}-slim AS app

# refresh global arguments
ARG WORKDIR
ARG USERNAME
ARG USER_UID
ARG USER_GID

# refresh PATH
ENV PATH=/opt/pipx/bin:${WORKDIR}/.venv/bin:$PATH \
 POETRY_VERSION=$POETRY_VERSION \
 PYTHONPATH=${WORKDIR} \
 # Don't buffer `stdout`
 PYTHONUNBUFFERED=1 \
 # Don't create `.pyc` files:
 PYTHONDONTWRITEBYTECODE=1

# ------------------------------ user management ----------------------------- #

RUN groupadd --gid $USER_GID "${USERNAME}" \
 && useradd --uid $USER_UID --gid $USER_GID -m "${USERNAME}"

# ------------------------------- app specific ------------------------------- #

WORKDIR ${WORKDIR}

RUN chown -R ${USER_UID}:${USER_GID} ${WORKDIR}

COPY --from=app-pre --chown=${USER_UID}:${USER_GID} ${WORKDIR} ${WORKDIR}

USER ${USERNAME}

ENTRYPOINT [ "python" ]
CMD [ "--version" ]


# ---------------------------------------------------------------------------- #
#                                   dev stage                                  #
# ---------------------------------------------------------------------------- #

FROM app-pre AS dev

# refresh global arguments
ARG WORKDIR
ARG USERNAME
ARG USER_UID
ARG USER_GID


USER root

# Add USERNAME to sudoers. Omit if you don't need to install software after connecting.
RUN apt-get update \
 && apt-get install -y sudo git iputils-ping wget \
 && echo ${USERNAME} ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/${USERNAME} \
 && chmod 0440 /etc/sudoers.d/${USERNAME}

USER ${USERNAME}

# install all remaning dependencies
RUN poetry install --no-root

USER ${USERNAME}

References

About docker and poetry

For more on users and groups

About renewing arguments at multiple stages

https://stackoverflow.com/a/53682110

Pyenv, Poetry and Pipx: three python tools to rule them all

2023-07-03T01:00:00+02:00

Update
Sharpen your axe
Pin those dependencies
Get control of your python environments
But first, what is a dependency?
A demonstration using Docker
Writing your project as a CLI with poetry
Building and publishing your project
Managing virtual environments inside docker

Update

I have formatted this entry into a workshop format for Som Energia, the cooperative I work with at Girona. You can find it here.

Sharpen your axe

If I had to chop down a tree in eight hours, I would spend six hours sharpening my axe.

This saying is usually attributed to Abraham Lincoln, who seemed very good at chopping down trees. The idea is that taking care of your tools will make your craft easier. This is true for any craft and specially of software development.

I wrote this as a short tutorial for a job I had. The idea was to introduce Poetry and dependency management to the people I worked with, and to move away from the requirements.txt file. These are the tools I use nowadays to manage my python projects and they have been working great for me. I will use Docker to show you how it all fits together, so you don't need to install anything.

Pin those dependencies

Pinned dependencies are important, as they help maintain and control functionalities of your project. Listing pinned dependencies ensures projects get outdated (i.e. they “rot”) in a manageable and understandable way.

There are some levels to how reproducible your project is:

Mention somewhere evident what version of Python your project is supposed to work with. A badge would suffice. See for example static badges, like this one .
Provide at least a requirements.txt with pinned dependencies.
Provide a setup.py file declaring dependencies.
Provide a pyproject.toml file in the best case
Use a dependency manager such as poetry, pipenv or hatch or conda.
Bundle everything in a Docker image.
Use multistage docker images to build your project.
Use a CI/CD tool to automate the process of building and deploying your project.

Get control of your python environments

Old is the xkcd comic about python environments. The gist is that isolated environments are important and you should use them. Even better if you know what python version you are using at all times and where is it installed. whereis python is your friend.

Same as with dependencies, there are some levels to how isolated your python environments are:

First level python -m venv .venv at your project level
Second level is to have pipx manage your system level utilities, i.e. normally CLI tools like cowsay or poetry or black
Second level is to let poetry manage your virtual environments
Third level is to let pyenv manage your python versions and your environments and let poetry just do dependency management
Fourth level is some python version manager that does not use shims and does not suck but I don't know of any as of today.
Fifth level is using Docker to build an isolated environment.

But first, what is a dependency?

A project may depend on another library: consider for example the pandas library. You normally install this with pip install pandas. That means that your project now has a dependency on pandas.
Pandas is a very large project and it has its own dependencies too. Depending on the complexity of the project, these may be hard to understand at first. For pandas, this may be very well listed as anaconda dependencies.
Each library at some point grows past the point where it needs to start tracking such growth through some kind of a versioning system. This is so that users (e.g. other developers like you) can write code depending on a specific expectated behaviours.
Sometimes a change of the version won't mean anything to your own code. Other times, however, it may introduce a change of its expected behaviour when running the same thing. This is loosely referred to as a breaking change. Consider the example for distutils telling its users how to migrate to new versions so their code won't break.

What is a virtual environment?

Make sure to read this https://realpython.com/python-virtual-environments-a-primer/ for more info. It's very complete and I won't try to improve on that.

Why do we need a virtual environment?

Virtual environment are there to keep you sane.
A virtual environment is a directory (a.k.a folder) holding a copy of python, isolated from other copies that you may have installed in your system.
Oftentimes such an environment is installed in a folder, generally called .venv
Any libraries installed with ./.venv/bin/python -m pip are linked to that version, i.e. they will be located under ./.venv/lib/pythonX.Y/site-packages/
Imagine the following weird example: You own three python projects, each one with its own dependencies
Library A has dependency X with version 1.0.0. Running something like python -m X.foo will yield the integer 42
Library B has dependency Y with version 2.0.0.
Library C has dependency X with version 3.0.0. Running something like python -m X.foo will yield None
You have now a dependency conflict: A and C have the same dependency, but with different versions
Library A does not support version 3.0.0 of dependency X, and vice versa
Can’t go installing and uninstalling versions every time I want to use A or C. I expect a 42 on the first case and a None on the second.
Solution: install two instances of python at different locations that can coexist because their dependencies are now isolated: we have a separated version of python installed per each project

The easy win: using `virtualenv`

virtualenv is the standard library. That means that without installing any other dependency, python -m venv .venv will create a copy of python within a folder called .venv. This folder will contain something like this:

.venv/
├── bin
├── include
├── lib
└── lib64 -> lib

And you normally "activate" it by calling the script inside /bin/activate.

The "a bit more involved" win using `pyenv` + `pipx` + `poetry`

In short, it means

Let pyenv manage versions of python. Follow their instructions to install it.
Create a virtual environment with pyenv virtualenv 3.9.10 <environment-name>.
Activate it with pyenv activate <environment-name>.
Let pyenv automatically activate an environment after entering a folder with pyenv local <environment-name>.
If you are running Windows, you can use pyenv-win
let pipx manage your system level utilities e.g. poetry
let poetry manage dependencies
install it system-wide with pipx install poetry
poetry creates virtual environments by default, we want to deactivate it
- poetry config virtualenvs.create false
- poetry config virtualenvs.in-project false

A demonstration using Docker

Spoiler: you will need docker. You can get free containers using play-with-docker.

Start by launching a new container named my-python-environment with the latest version of ubuntu and run a bash shell. The --rm flag will remove the container after you exit it so it won't clutter your system.

docker run -it --name my-python-environment --rm ubuntu:latest bash`

You are now starting from a clean slate. Let's install some dependencies.

apt update && apt install curl git vim build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev curl \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev -y
curl https://pyenv.run | bash

You will now have pyenv installed. Let's configure it. As per the instructions, we need to add the following to our ~/.bashrc file:

vim ~/.bashrc

Go to the end of the file and add the following:

# Load pyenv automatically by appending
# the following to
#~/.bash_profile if it exists, otherwise ~/.profile (for login shells)
# and ~/.bashrc (for interactive shells) :

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Save and quit. Now we need to refresh our shell to load the new configuration:

exec "$SHELL"

You can now use pyenv to install a new version of python. Let's install version 3.10.9:

pyenv install 3.10.9

Let's make it available globally:

pyenv global 3.10.9

We can also install different versions of python and switch between them:

pyenv install 3.8.12

We have now python 3.10.9 installed globally. Let's install pipx with it:

python3 -m pip install --upgrade pip
python3 -m pip install --user pipx
python3 -m pipx ensurepath

Again, we need to refresh our shell to load the new configurations:

exec "$SHELL"

Let's install poetry with pipx:

pipx install poetry==1.4.2 --force

And confirm it's installed with:

$ poetry --version
Poetry (version 1.4.2)
$ whereis poetry
poetry: /root/.local/bin/poetry

We want to configure poetry to not create virtual environments by default:

# configure poetry
poetry config virtualenvs.create false`
poetry config virtualenvs.in-project false`

We can create a new project with poetry named mypackage:

# create project
mkdir ~/mypackage
cd ~/mypackage

We will now create a virtual environment for this project only

pyenv virtualenv 3.10.9 myenv
pyenv activate myenv
pyenv rehash # refresh pyenv shims

use poetry dialog to create new project

poetry init

The dialog will ask you a few questions. You can skip them by pressing enter. The dialog will then create a pyproject.toml file. You can now manage dependencies with poetry:

poetry add pandas

and remove dependencies

poetry remove pandas

Every time you add or remove packages with poetry, two things happen:

the pyproject.toml file is updated and a new line is added or deleted under the dependencies section
a poetry.lock file is created or updated, containing the exact versions of the dependencies you are using and their relations.

Let's install cowsay with poetry:

poetry add cowsay
poetry add --group dev black
pyenv rehash # to update shims
cowsay hola # a cow should greet you, in spanish
  ____
| hola |
  ====
    \
     \
       ^__^
       (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||

Writing your project as a CLI with `poetry`

cd ~/mypackage
mkdir mypackage
touch mypackage/__init__.py
touch mypackage/cli.py

Open mypackage/cli.py and add the following:

import cowsay

def cli():
    return cowsay.cow('Hello World')

if __name__ == "__main__":
    cli()

Confirm that your program works by running it with python mypackage/cli.py:

$ python mypackage/cli.py
___________
| Hello World |
  ===========
           \
            \
              ^__^
              (oo)\_______
              (__)\       )\/\
                  ||----w |
                  ||     ||

We can update our pyproject.toml file to include a CLI entrypoint:

[tool.poetry]
name = "mypackage"
version = "0.1.0"
description = "some description"
authors = ["Your Name <youremail@yopmail.com>"]
readme = "README.md"
packages = [{include = "mypackage"}] # new

[tool.poetry.dependencies]
python = "^3.10"
pandas = "^2.0.0"
cowsay = "^5.0"

[tool.poetry.scripts]
mypackage-cli = "mypackage.cli:cli" # new

[tool.poetry.group.dev.dependencies]
black = "^23.3.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

And now we can install our package with poetry and run our little CLI by calling mypackage-cli:

poetry install  # install package along with its CLI
pyenv rehash # refresh python shims
mypackage-cli # launch cli

Building and publishing your project

Make sure you have a valid account at pypi. You can create one here.

Run poetry build (at the level where a valid pyproject.toml is present). From this point on, you have two options:

Publish to pypi

Run poetry publish --username myprivaterepo --password <password> --repository pypi

Publish to a private repository

if you wish to publish to a private repository, say, pypi.myprivaterepo.org:

Add repository to the config in poetry, with poetry config repositories.myprivaterepo https://pypi.myprivaterepo.org/
Careful to not add the /simple bit if your privare repo is using pypiserver, see https://github.com/pypiserver/pypiserver/issues/329#issuecomment-688883871
now your private pypi ****repository is aliased with myprivaterepo

Now you can run poetry publish --username myprivaterepo --password <password> --repository myprivaterepo. See https://python-poetry.org/docs/libraries#publishing-to-pypi for more documentation

Creating your own private repository using docker compose

Launch your own pypi repository with https://github.com/pypiserver/pypiserver and https://github.com/pypiserver/pypiserver/blob/master/docker-compose.yml

Adding dependencies from private repositories to pyproject.toml

You may also want to add dependencies from private repositories. These repos normally need keys to access them. Make sure to follow the instructions from your private repository to add credentials to your pyproject.toml file. Generally, the process is as follows:

Add a source to pyproject.toml file
poetry source add myprivaterepo https://pypi.myprivaterepo.org
This should modify your pyproject.toml file and will add something like this

toml [[tool.poetry.source]] name = "myprivaterepo" url = "https://pypi.myprivaterepo.org/" default = false secondary = false
Add credentials for that repository
poetry config http-basic.myprivaterepo <user> <password>
Add dependencies using the --source argument

bash poetry add --source myprivaterepo my-private-package=0.2.1

Managing virtual environments inside docker

Last but not least. People tend to argue over such scenario. Is isolation within isolation necessary? Is a virtual environment needed inside a docker container? See

Answer is “it depends”, but it gives more control over dependencies and their state. I prefer it using docker multistage builds. I will cover this in a next post hopefully soon.

Until next time!

Using dbt, fal and novu for forecasting pipelines

2023-07-03T00:00:00+02:00

I did a lightning talk at the pydata Barcelona 2023 about using dbt and fal on behalf of Som Energia. That was my first talk in a while and I was a bit nervous, but I think it went well. I'm happy to have been able to share some of the things I've learned in the last few months.

You can find the repository with the slides and the code here

Make sure to check out the other talks, there were some really interesting ones!

Pelican (and personal) update

2023-03-12T13:14:00-06:00

Somehow inbetween my last update and now, the deployment workflow of github sites changed toward Github Actions and it broke my site. I managed to implement a new and brittle workflow that works and makes me happy that I can write stuff again. I also bought a domain at 301621.xyz and I'm using it for this site at bluesmonk.301621.xyz. I'm working towards a better CI on https://github.com/diegoquintanav/pelican-to-github-pages at the moment.

Hug a rock today

2022-01-01T13:14:00-06:00

I happen to enjoy the collection of A very short introduction books. It's a bargain (used, they cost between 3 and 10 euros), and most of the times you get very good introductions to a topic. Sometimes they are not very good, and sometimes they leave you wanting to write about something, like the Information book by Luciano Floridi.

On the subject of the ethics of information, he quotes a passage in one of Albert Einstein's letters, in response to a girl who was in the process of grieving the death of her sister.

A human being is part of a whole, called by us the “Universe,” a part limited in time and space. He experiences himself, his thoughts and feelings, as something separated from the rest — a kind of optical delusion of his consciousness. This delusion is a kind of prison for us, restricting us to our personal desires and to affection for a few persons nearest us. Our task must be to free ourselves from this prison by widening our circles of compassion to embrace all living creatures and the whole of nature in its beauty. Nobody is capable of achieving this completely, but the striving for such achievement is in itself a part of the liberation and a foundation for inner security.

Then he quotes a passage from Lynn Townsend White Jr.:

Do people have ethical obligations toward rocks? (...) If the time comes when any considerable group of us such a question is no longer ridiculous, we may be on the verge of a change of value structures that will make possible measures to cope with growing ecologic crisis. One hopes that there is enough time left.

And then he concludes that this is the right approach to the ethics of information, from an ecologist perspective.

I enjoyed this book a lot. And if you happen to come across this post, I'd suggest you give it a read.

Modeling the Pinochet Dataset using Neo4j

2020-09-09T13:14:00-06:00

Introduction

On September 11, 1973, Chile faced a military coup lead by Augusto Pinochet and the US agenda. Pinochet would be then dictator and leader of the military regime until 1989. During the regime, thousands of people were kidnapped, tortured and killed. The atrocities committed by the regime are a scar that defines what Chile is today, and there are families that are still looking for clues to find their missing relatives. "Nostalgia for the light" is a relevant documentary portraying people looking for their relatives, comparing it to the way astronomers look for stars in the sky.

As part of the efforts to shed light on the truth about what happened, two official reports can be noted:

The first one was issued under the presidency of Patricio Alwyin in the year 1991, shortly after the military dictatorship ended. This document is known as the Rettig Report.
The second report was issued during the first presidency of Michelle Bachelet, and it attempts to extend these efforts in what is known as the Valech Report.

It came to my knowledge that some people, more specifically, Danilo Freire and others, published a dataset and an article containing the information about the 903 pages of the report, along with a Github repository.

Freire, Danilo, David Skarbek, John Meadowcroft, and Eugenia Guerrero. 2019. “Deaths and Disappearances in the Pinochet Regime: A New Dataset.” SocArXiv. May 31. doi:10.31235/osf.io/vqnwu.

From the article that goes with the dataset,

Although the (Rettig) Report is a valuable source of information for scholars, quantitative researchers cannot readily use the rich data it contains. In this paper, we present a manually-coded dataset with all information from the Truth Commission Report plus new variables we collected to complement the original records. We transcribed every personal detail from the 903 pages of the English translation of the Report, assigned a unique identification number to each victim, then georeferenced the location of the human rights abuses. We added coordinates of latitude and longitude to every location mentioned in the report, such as places used to torture, interrogate or murder the victims.

The repository explains this better

This Github repository contains data and documented R code for Deaths and Disappearances in the Pinochet Regime: A New Dataset by Freire et al (2019). We coded the personal details of 2,398 victims named in the Chilean Truth Commission Report along with information about the perpetrators and geographical coordinates for all identifiable atrocity locations. The dataset covers from 1973 to 1990 and includes 59 indicators. Please refer to our accompanying article and our online appendix for more details.

map of events from the pinochet repository

I wanted to model this dataset as a graph, with the intention to explore the data as what it is, a network that links victims to places and events. This an attempt to learn and to remember.

Roadmap

I will be first modeling the dataset as graph, using neo4j and py2neo. I will be using docker, as always. Next I will serve this graph using graphql through python's graphene library (although I recommend using apollo.js instead), and finally I will consume this service using a leaflet.js map.

I have other ideas, but I have to finish what I have right now. I would like to link this data to other information related to this dataset (e.g. the Valech report and the Londres 38 site, which we will see later on), perhaps through the graphql layer, or maybe using a semantic model and tools like ontop and protege. I learnt about these technologies the past semester and I've been itching to use them in something interesting.

Before starting

Tabular form data exploration

You can see an analysis of the data in tabular form using Pandas and a rudimentary map implementation using folium, in this notebook.

The notebooks is as of now in spanish only, and it will take a while to translate it. You can also see the analysis done in R in the original repository!

I've used these tutorials up to some extent. Please check them if you wish to have more code examples!

On graphql

Check these resources if you don't know what is this graphql thing.

https://www.howtographql.com/, a very good intro to graphql
https://dev.to/bogdanned/the-graph-in-graphql-1l99, an in-depth article about the graphql specifications
https://www.apollographql.com/blog/graphql-explained-5844742f195e/

On neo4j

I'm using neo4j version 3.5 from the docker images, for no reason in particular apart that I've used it before. I may update to 4.x in the future, if I find reasons to do so.

Modeling

First approach: Don't change anything

The dataset contains informations about perpetrators, victims, violence events and event locations. We will develop models around these concepts, and we will stablish relationships between them afterwards.

First we will extend the GraphObject used in py2neo to adapt it to our needs.

class BaseModel(GraphObject):
    """Extends the GraphObject class with utility methods"""

    def __init__(self, **kwargs):
        for key, value in kwargs.items():
            if hasattr(self, key):
                setattr(self, key, value)

    @classmethod
    def all(cls):
        return cls.match(graph)

    def as_dict(self):
        return dict(self.__node__)

    def fetch(self):
        pk = getattr(self, "__primarykey__")
        return self.match(graph, getattr(self, pk)).first()

    def fetch_by_attr(self, attr, value, exact=True):
        # https://py2neo.org/v4/ogm.html#object-matching
        operator = "=" if exact else "=~"
        q = f"_.{attr} {operator} '{value}'"  #  e.g. _.name =~ ".*K.*" noqa
        return list(self.match(graph).where(q))

Concepts (or Nodes)

Perpetrators

Let's start with the Perpetrator model, the easiest. From the codebook we see that they have the following properties

perpetrator_affiliation, string
perpetrator_affiliation_detail, string
war_tribunal, boolean

This is easily implemented as

class Perpetrator(BaseModel):

    perpetrator_affiliation = Property()
    perpetrator_affiliation_detail = Property()
    war_tribunal = Property()

Victims

individual_id, integer
group_id, int
first_name, string
last_name, string
age, string: (this is to support NAs in the data, but they should be handled better e.g. with native nulls in neo4j, I agree. See https://stackoverflow.com/questions/40788440/neo4j-use-merge-with-null-values)
minor, boolean
male, boolean
number_previous_arrests, string (same as in age)
occupation, string
occupation_detail, string
victim_affiliation, string
victim_affiliation_detail, string
targeted, string

class Location(BaseModel):

    exact_coordinates = Property()
    location = Property()
    place = Property()
    location_order = Property()
    latitude = Property()
    longitude = Property()

ViolentEvent

Assuming each row describes a violent event, we say that a ViolentEvent develops in different locations specified by the location_n attribute.

A ViolentEvent has thus the following node properties:

violence, string
method, string
interrogation, boolean
torture, boolean
mistreatment, boolean
press, boolean
start_date_daily, date or string
end_date_daily, date or string
start_date_monthly, date or string
end_date_monthly, date or string

There are up to 6 locations per event, according to the codebook. We will associate a violent event to a series of locations, like

series of locations associated to one violent event

Where location_n describes the n-th location from a series of events. From the codebook, we observe that

exact_coordinates_n : We matched the event sites with coordinates of latitude and longitude. As the report does not have the precise location of all events, we used the closest reference available. This is a dummy variable stating whether coordinates are precise (street level) or not. 1 = yes. There are six variables in the dataset, each pertaining to one location where the individual was found or taken to.
location_n : Where the individual was seen or found. There are up to 6 locations, so we coded them as location_1 to location_6 . The same pattern repeats in the variables below. The compilation of the location_n variable was based completely on information given in the Truth Report. However, since this information was in a string format (e.g. intersection of Calle Grecia and Avenida Rosa), creating a new variable incorporating each location’s latitude and longitude was necessary to pursue further analysis of the trends in deaths and disappearances. The format chosen was decimal coordinates.
place_n : Place where the individual was spotted/reported to be seen. (in chronological order, from 1 to 6 places). - Categories: Home; Work; University; In custody; In public; In hospital; Unknown

These locations look like this in the original dataset

First rows of the dataset, slicing on the location related attributes

Relationships

We link nodes using RelatedTo and RelatedFrom from py2neo's API. For example, we can link Location and ViolentEvent as follows

class Location(BaseModel):

    # properties omitted

    # outgoing relationships
    next_location = RelatedTo("Location", "NEXT_LOCATION")

    # incoming relationships
    previous_locations = RelatedFrom("Location", "NEXT_LOCATION")
    first_violent_events = RelatedFrom("ViolentEvent", "FIRST_LOCATION")
    in_violent_events = RelatedFrom("ViolentEvent", "IN_LOCATION")
    last_violent_events = RelatedFrom("ViolentEvent", "LAST_LOCATION")

class ViolentEvent(BaseModel):

    # properties omitted

    # outgoing relationships
    first_location = RelatedTo("Location", "FIRST_LOCATION")
    in_location = RelatedTo("Location", "IN_LOCATION")
    last_location = RelatedTo("Location", "LAST_LOCATION")

    # incoming relationships
    victims = RelatedFrom("Victim", "VICTIM_OF")
    perpetrators = RelatedFrom("perpetrator", "PERPETRATOR_OF")

From the rules mentioned above, we end up with a graph that has this shape

shape of the final graph

Polishing things

The first approach is a bit flaky. We observe that we don't have a proper id or primary keys. These are not explicitly provided in the dataset, so we have to either create them artificially or pick one attribute from each node that will act as one. We can observe how this affects our final graph if we query entities with the same attributes.

Take for example all the locations that refer to La Legua, a Población or shantytown in the south of Santiago that has a powerful history of struggle linked to political conflicts and lately to drug trafficking and crime, as in the words of Rodrigo Ganter (2007, in spanish only).

La Legua has likely consolidated as a recognizable entity within the urban weave of Santiago because of the superposition of segregation, political conflicts and a disciplined culture. The structure of its community and its culture have been what have marked its boundaries with precision, rather than its spatial reality

http://www.plataformaurbana.cl/archive/2009/09/12/la-legua-emergencia-incisiones-urbanas-contra-el-narcotrafico/#identifier_0_10693

La Legua is known for being one of the places that presented resistance from the first day to the militar strike, which along with its history made it a target during the regime. In our graph database, we can observe the nodes that represent this location:

$ MATCH (n:Location) where n.location =~ '.*Legua.*' return n LIMIT 5

╒══════════════════════════════════════════════════════════════════════╕
│"n"                                                                   │
╞══════════════════════════════════════════════════════════════════════╡
│{"location":"La Legua shantytown","place":"In Public","location_order"│
│:"1","latitude":"-33.48722","longitude":"-70.63556"}                  │
├──────────────────────────────────────────────────────────────────────┤
│{"location":"La Legua shantytown","place":"In Public","location_order"│
│:"1","latitude":"-33.48722","longitude":"-70.63556"}                  │
├──────────────────────────────────────────────────────────────────────┤
│{"location":"La Legua shantytown","place":"In Public","location_order"│
│:"1","latitude":"-33.48722","longitude":"-70.63556"}                  │
├──────────────────────────────────────────────────────────────────────┤
│{"location":"La Legua shantytown","place":"In Public","location_order"│
│:"1","latitude":"-33.48722","longitude":"-70.63556"}                  │
├──────────────────────────────────────────────────────────────────────┤
│{"location":"La Legua shantytown","place":"In Public","location_order"│
│:"1","latitude":"-33.48722","longitude":"-70.63556"}                  │
└──────────────────────────────────────────────────────────────────────┘

We observe that although these are different nodes, they have the same attributes. The problem here is that we are loading repeated nodes in our database. We want to resolve these entities as equivalent at insertion time, to merge them into one single node.

Five nodes representing the same thing. We can merge them into one node.

From the py2neo documentation, we could achieve this by using an existing attribute, like

class Location(BaseModel):

    __primarykey__ = "location"  # the name of the location
    # rest of model below

But we will soon realize that we may be ignoring those locations that have equal location attribute but are different in other attributes. Take for instance the following query:

MATCH (n:Location) where n.location =~ '.*Legua.*' return n.location, n.exact_location, n.place, n.latitude, n.longitude, n.location_order LIMIT 25


╒═════════════════════╤══════════════════╤═══════════╤════════════╤═════════════╤══════════════════╕
│"n.location"         │"n.exact_location"│"n.place"  │"n.latitude"│"n.longitude"│"n.location_order"│
╞═════════════════════╪══════════════════╪═══════════╪════════════╪═════════════╪══════════════════╡
│"La Legua shantytown"│null              │"In Public"│"-33.48722" │"-70.63556"  │"1"               │
├─────────────────────┼──────────────────┼───────────┼────────────┼─────────────┼──────────────────┤
│"La Legua shantytown"│null              │"In Public"│"-33.48722" │"-70.63556"  │"1"               │
├─────────────────────┼──────────────────┼───────────┼────────────┼─────────────┼──────────────────┤
│"La Legua shantytown"│null              │"In Public"│"-33.48722" │"-70.63556"  │"1"               │
├─────────────────────┼──────────────────┼───────────┼────────────┼─────────────┼──────────────────┤
│"La Legua shantytown"│null              │"In Public"│"-33.48722" │"-70.63556"  │"1"               │
├─────────────────────┼──────────────────┼───────────┼────────────┼─────────────┼──────────────────┤
│"La Legua shantytown"│null              │"In Public"│"-33.48722" │"-70.63556"  │"1"               │
├─────────────────────┼──────────────────┼───────────┼────────────┼─────────────┼──────────────────┤
│"La Legua shantytown"│null              │"In Public"│"-33.48722" │"-70.63556"  │"1"               │
├─────────────────────┼──────────────────┼───────────┼────────────┼─────────────┼──────────────────┤
│"La Legua shantytown"│null              │"Home"     │"-33.48722" │"-70.63556"  │"1"               │
├─────────────────────┼──────────────────┼───────────┼────────────┼─────────────┼──────────────────┤
│"La Legua shantytown"│null              │"Home"     │"-33.48722" │"-70.63556"  │"1"               │
├─────────────────────┼──────────────────┼───────────┼────────────┼─────────────┼──────────────────┤
│"La Legua shantytown"│null              │"In Public"│"-33.48722" │"-70.63556"  │"1"               │
├─────────────────────┼──────────────────┼───────────┼────────────┼─────────────┼──────────────────┤
│"La Legua shantytown"│null              │"Home"     │"-33.48722" │"-70.63556"  │"1"               │
└─────────────────────┴──────────────────┴───────────┴────────────┴─────────────┴──────────────────┘

We see that although these nodes share the same latitude and longitude, some events occured In public where as other events occured at the victim's home.

In order to resolve this, we will hash these attributes and use the result as a key to avoid redundant data. In other words, we will use and MD5 hash function from python's hashlib

>>> hashlib.md5(b"foo").hexdigest()
'acbd18db4cc2f85cedef654fccc4a4d8'

In code, this would be something like

import hashlib
import typing as T

def to_md5(elements: T.List[str], *args, **kwargs) -> str:
    """calculates the MD5 hash of the string of concatenated elements"""
    pre = "".join(str(x).lower() for x in elements)
    return hashlib.md5(pre.encode("utf-8")).hexdigest()

I'm not thinking too much into this: I will produce at least 2398 different keys, and considering this is not for cryptographic purposes, MD5 gives me plenty of room for that.

For tables and graphs that are intended to grow over time, you should think more about using this approach, for many reasons. MD5 is full of holes, it's not secure for cryptographic purposes like storing passwords (its initial purpose), and it has known hash collision issues.

For the case of Perpetrator we note that from 2398 total instances in the db, we have these unique entities:

$ MATCH (n:Perpetrator)  RETURN COUNT(DISTINCT n.perpetrator_affiliation_detail)
228
$ MATCH (n:Perpetrator)  RETURN COUNT(DISTINCT n.perpetrator_affiliation)
3
$ MATCH (n:Perpetrator)  RETURN COUNT(DISTINCT n.war_tribunal)
2

We can create 228*3*2 = 1392 unique ids if we concatenate attribute values. We can compress this index to have some meaning, like a hash of (attr1 + attr2 + attr3). As it is, Perpetrator is sort of a junk dimension in the Data Warehousing jargon.

class Perpetrator(BaseModel):

    __primarykey__ = "perpetrator_id"

    perpetrator_id = Property()
    # rest of model below

For the case of ViolentEvents it is more difficult. We also have 2398 instances, but how do we tell them apart? Let's have a look at the dates, which seem promising as primary keys.

MATCH (n:ViolentEvent) RETURN COUNT(DISTINCT n.start_date_daily), COUNT(DISTINCT n.end_date_daily), COUNT(DISTINCT n.start_date_monthly), COUNT(DISTINCT n.end_date_monthly)

╒═════════╤═════════╤═════════╤═════════╕
│"count(di│"count(di│"count(di│"count(di│
│stinct n.│stinct n.│stinct n.│stinct n.│
│start_dat│end_date_│start_dat│end_date_│
│e_daily)"│daily)"  │e_monthly│monthly)"│
│         │         │)"       │         │
╞═════════╪═════════╪═════════╪═════════╡
│652      │658      │151      │155      │
└─────────┴─────────┴─────────┴─────────┘

The main difficulty is with the data as we have it right now, we have no way to know that two nodes refer to the same event. We will assume that all instances in this table are unique i.e. all violent events are unique, and we will give them a UUID accordingly.

class ViolentEvent(BaseModel):

    __primarykey__ = "event_uuid"

    event_uuid = Property()
    # rest of model goes here

With these keys in place, we start over and recreate the graph. We check for locations referring to La Legua, and see that now there are three instances in total. We removed a lot of redundant data this way!

MATCH (n:Location) where n.location =~ '.*Legua.*' return n LIMIT 25

╒══════════════════════════════════════════════════════════════════════╕
│"n"                                                                   │
╞══════════════════════════════════════════════════════════════════════╡
│{"location":"La Legua shantytown","place":"In Public","location_order"│
│:"1","location_id":"cb95da6b208a47984cb25051d4ec9561","latitude":"-33.│
│48722","longitude":"-70.63556"}                                       │
├──────────────────────────────────────────────────────────────────────┤
│{"location":"La Legua","place":"In Public","location_order":"1","locat│
│ion_id":"a25c89cb84f42e76be3c60c34d536abd","latitude":"-33.48722","lon│
│gitude":"-70.63556"}                                                  │
├──────────────────────────────────────────────────────────────────────┤
│{"location":"La Legua shantytown","place":"Home","location_order":"1",│
│"location_id":"8a0bf7de20dd6ed6c3f76aab95d942bb","latitude":"-33.48722│
│","longitude":"-70.63556"}                                            │
└──────────────────────────────────────────────────────────────────────┘

In the same way, we see that

$ MATCH (n:Perpetrator)  RETURN COUNT(DISTINCT n)
244

Loading the dataset into a graph

We are loading the dataset with one script, it's long so please bear with me. You can check the full script in github

import os
import csv
import typing as T

from py2neo import Graph

graph = Graph(
    host=os.environ.get("NEO4J_HOST"),
    port=os.environ.get("NEO4J_PORT"),
    user=os.environ.get("NEO4J_USER"),
    password=os.environ.get("NEO4J_PASSWORD"),
)

def clear_graph():
    return graph.run("MATCH (n) DETACH DELETE n").stats()

def __create_node(graph_object, dry_run=False):
    current_app.logger.debug(f"Creating {graph_object}")
    graph.create(graph_object)
    current_app.logger.debug("Done")

def get_locations(row: dict) -> T.List[Location]:
    """Parses the locations in the dataset into an ordered collection of Location instances"""

    locations = []

    # there are 6 locations max in the dataset
    for n in range(1, 7):
        # We will fetch those that are not empty or not "NA"
        location_name = row.get(f"location_{n}", None)
        if location_name is not None and location_name != "NA":
            loc_mapping = {
                "exact_location": row[f"exact_coordinates_{n}"],
                "location": location_name,
                "place": row[f"place_{n}"],
                "latitude": row[f"latitude_{n}"],
                "longitude": row[f"longitude_{n}"],
                "location_order": n,
            }

            # instance of Location, using the MD5 hash of attributes as ID
            loc = Location(
                location_id=to_md5(loc_mapping.values()),
                **loc_mapping
            )

            locations.append(loc)
    return locations

def seed_graph(filepath, **kwargs):

    with open(filepath) as fp:
        csv_reader = csv.DictReader(f=fp)
        for row in csv_reader:
            # create victims
            victim = Victim(
                individual_id=row["individual_id"],  # pk
                group_id=row["group_id"],
                first_name=row["first_name"],
                last_name=row["last_name"],
                age=row["age"],
                minor=row["minor"],
                male=row["male"],
                nationality=row["nationality"],
                number_previous_arrests=row["number_previous_arrests"],
                occupation=row["occupation"],
                occupation_detail=row["occupation_detail"],
                victim_affiliation=row["victim_affiliation"],
                victim_affiliation_detail=row["victim_affiliation_detail"],
                targeted=row["targeted"],
            )

            perp_mapping = {
                "perpetrator_affiliation": row["perpetrator_affiliation"],
                "perpetrator_affiliation_detail": row[
                    "perpetrator_affiliation_detail"
                ],
                "war_tribunal": row["war_tribunal"],
            }

            # obtain the md5 as PK
            perp = Perpetrator(
                perpetrator_id=to_md5(perp_mapping.values()), **perp_mapping
            )

            locations = get_locations(row)

            event = ViolentEvent(
                event_id=str(uuid.uuid4()),  # pk
                violence=row["violence"],
                method=row["method"],
                interrogation=row["interrogation"],
                torture=row["torture"],
                mistreatment=row["mistreatment"],
                press=row["press"],
                start_date_daily=row["start_date_daily"],
                end_date_daily=row["end_date_daily"],
                start_date_monthly=row["start_date_monthly"],
                end_date_monthly=row["end_date_monthly"],
                page=row["page"],
                additional_comments=row["additional_comments"],
            )

            # create relationships between event and locations
            if locations:
                event.first_location.add(locations[0])
                event.last_location.add(locations[-1])
                for index, location in enumerate(locations):
                    event.in_location.add(location)
                    if index != 0:
                        locations[index - 1].next_location.add(location)

            victim.victim_of.add(event)
            perp.perpetrator_of.add(event)

            # create nodes
            __create_node(victim, dry_run)
            __create_node(perp, dry_run)
            __create_node(event, dry_run)

            for location in locations:
                __create_node(location, dry_run)

We could have done something using Cypher and APOC, a library for many kinds of tasks within neo4j. At some point though, Cypher's capabilities fell short so I had to rely on python.

After we've done this, we can start querying the graph with things like

General statistics

from https://neo4j.com/developer/kb/how-do-i-produce-an-inventory-of-statistics-on-nodes-relationships-properties/ we can create an inventory of miscellaneous statistics of the graph.

MATCH (n)
WITH labels(n) as labels, size(keys(n)) as props, size((n)--()) as degree
RETURN
DISTINCT labels,
count(*) AS NumofNodes,
avg(props) AS AvgNumOfPropPerNode,
min(props) AS MinNumPropPerNode,
max(props) AS MaxNumPropPerNode,
avg(degree) AS AvgNumOfRelationships,
min(degree) AS MinNumOfRelationships,
max(degree) AS MaxNumOfRelationships



╒════════╤════════╤════════╤════════╤════════╤════════╤════════╤════════╕
│"labels"│"NumofNo│"AvgNumO│"MinNumP│"MaxNumP│"AvgNumO│"MinNumO│"MaxNumO│
│        │des"    │fPropPer│ropPerNo│ropPerNo│fRelatio│fRelatio│fRelatio│
│        │        │Node"   │de"     │de"     │nships" │nships" │nships" │
╞════════╪════════╪════════╪════════╪════════╪════════╪════════╪════════╡
│["Victim│2398    │13.0    │13      │13      │1.0     │1       │1       │
│"]      │        │        │        │        │        │        │        │
├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
│["Violen│2398    │13.0    │13      │13      │4.690992│1       │9       │
│tEvent"]│        │        │        │        │49374478│        │        │
│        │        │        │        │        │1       │        │        │
├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
│["Locati│1841    │6.0     │6       │6       │5.652906│2       │227     │
│on"]    │        │        │        │        │02933188│        │        │
│        │        │        │        │        │8       │        │        │
├────────┼────────┼────────┼────────┼────────┼────────┼────────┼────────┤
│["Perpet│244     │4.0     │4       │4       │1.0     │1       │1       │
│rator"] │        │        │        │        │        │        │        │
└────────┴────────┴────────┴────────┴────────┴────────┴────────┴────────┘

We observe right away that on average, a violent event is linked to ~4.7 nodes. We would like to be more specific, and for this we will craft the subgraph that we would like to match, and then we will get the outgoing degree of the event for all victims.

This is equivalent to asking for the length on average of the paths walked for a victim. This boils down to fetch the initial and the last location of each event, and count how many locations are in the path formed by those locations.

match (v:Victim)-[:VICTIM_OF]->(e:ViolentEvent)
with e
match pathA=(fl:Location)<-[:FIRST_LOCATION]-(e)-[:LAST_LOCATION]->(ll:Location)
with fl, ll
match pathB=(fl)-[:NEXT_LOCATION*]->(ll)
with length(pathB) as pathLength
return avg(pathLength)

╒══════════════════╕
│"avg(pathLength)" │
╞══════════════════╡
│1.4383708467309761│
└──────────────────┘

In other words, a victim visits ~1.44 places on average before ending its journey (either by getting killed, dissapeared, or suicide, for example).

This is the same as above, but with substracting -2 to the length because this path contains the event twice each time

match (e:ViolentEvent)
with e
match path=(e)-[:FIRST_LOCATION]->(fl:Location)-[:NEXT_LOCATION*]->(ll:Location)<-[:LAST_LOCATION]-(e)
with length(path) - 2 as pathSize
return avg(pathSize)

╒══════════════════╕
│"avg(pathSize)"   │
╞══════════════════╡
│1.4383708467309761│
└──────────────────┘

Using the geospatial module of neo4j

From https://neo4j.com/docs/cypher-manual/3.5/functions/spatial/, we could update relationships with their geodesic distance. For now we just want to see it.

MATCH (v:Victim)-[:VICTIM_OF]->(e:ViolentEvent)
WITH e
MATCH pathA=(fl:Location)<-[:FIRST_LOCATION]-(e)-[:LAST_LOCATION]->(ll:Location)
WITH fl, ll
MATCH pathB=(fl)-[:NEXT_LOCATION*]->(ll)
WITH point({x: toFloat(fl.longitude) , y: toFloat(fl.latitude), crs:'wgs-84'}) as p1, point({x: toFloat(ll.longitude), y: toFloat(ll.latitude), crs:'wgs-84'}) as p2, fl, ll
RETURN fl.location, ll.location, distance(p1, p2)
LIMIT 5

╒═══════════════════╤═══════════════════╤══════════════════╕
│"fl.location"      │"ll.location"      │"distance(p1, p2)"│
╞═══════════════════╪═══════════════════╪══════════════════╡
│"Ochagavia Bridge o│"Ochagavia Bridge o│0.0               │
│ver Callejón Lo Ova│ver Callejón Lo Ova│                  │
│lle"               │lle"               │                  │
├───────────────────┼───────────────────┼──────────────────┤
│"Intersection of Lo│"On route to Barros│7058.819005808001 │
│ Sierra and Lo Espe│ Lucas Hospital "  │                  │
│jo"                │                   │                  │
├───────────────────┼───────────────────┼──────────────────┤
│"downtown Santiago"│"Villa Grimaldi"   │12154.56171338133 │
├───────────────────┼───────────────────┼──────────────────┤
│"downtown Santiago"│"Villa Grimaldi"   │12154.56171338133 │
├───────────────────┼───────────────────┼──────────────────┤
│"Calle Nuble, 1034"│"Central Emergency │2995.4570977906865│
│                   │Clinic"            │                  │
└───────────────────┴───────────────────┴──────────────────┘

From the second result, we can check with google maps that the distance approximatedly 9 kilometers

From "Ñuble 1034" to the "Central Emergency Clinic" it's a 3 km distance, approximately.

Which type of events are linked to more locations?

MATCH (v:Victim)-[:VICTIM_OF]->(e:ViolentEvent)
WITH e
MATCH path=(e)-[r:IN_LOCATION]->(l:Location)
RETURN e.violence, count(l) AS degree
ORDER BY degree DESC

╒═════════════════════════════════════╤════════╕
│"e.violence"                         │"degree"│
╞═════════════════════════════════════╪════════╡
│"Killed"                             │2331    │
├─────────────────────────────────────┼────────┤
│"Disappearance"                      │1438    │
├─────────────────────────────────────┼────────┤
│"Disappearance, Information of Death"│212     │
├─────────────────────────────────────┼────────┤
│"Unresolved"                         │86      │
├─────────────────────────────────────┼────────┤
│"Suicide"                            │16      │
├─────────────────────────────────────┼────────┤
│"NA"                                 │2       │
└─────────────────────────────────────┴────────┘

If we analyze the average number of locations, we get a different perspective

MATCH (v:Victim)-[:VICTIM_OF]->(e:ViolentEvent)
WITH e
MATCH path=(e)-[r:IN_LOCATION]->(l:Location)
WITH e, count(l) as degree
RETURN e.violence, avg(degree) AS avgDegree
ORDER BY avgDegree DESC

╒═════════════════════════════════════╤══════════════════╕
│"e.violence"                         │"avgDegree"       │
╞═════════════════════════════════════╪══════════════════╡
│"Suicide"                            │2.0               │
├─────────────────────────────────────┼──────────────────┤
│"NA"                                 │2.0               │
├─────────────────────────────────────┼──────────────────┤
│"Disappearance, Information of Death"│1.9629629629629628│
├─────────────────────────────────────┼──────────────────┤
│"Killed"                             │1.8618210862619833│
├─────────────────────────────────────┼──────────────────┤
│"Disappearance"                      │1.7388149939540511│
├─────────────────────────────────────┼──────────────────┤
│"Unresolved"                         │1.3230769230769233│
└─────────────────────────────────────┴──────────────────┘

On average, people that died because of suicide visited more locations than those that were killed or dissapeared. It's hard to consider this fact as final, since there is that "NA" messing with the data.

Longest paths

We want to know about those victims that moved the most, i.e. have the longest trajectories. We can do so with

MATCH (v:Victim)-[:VICTIM_OF]->(e:ViolentEvent)
WITH v, e
MATCH pathA=(fl:Location)<-[:FIRST_LOCATION]-(e)-[:LAST_LOCATION]->(ll:Location)
WITH v, e, fl, ll
MATCH pathB=(fl)-[:NEXT_LOCATION*]->(ll)
WITH v, e, pathB, length(pathB) as pathLength
RETURN v, e, pathB
ORDER BY pathLength DESC
LIMIT 1

This result corresponds to Jose Manuel Díaz Inostroza, a 29 years old farm worker that was tortured and killed after one month since he was taken, along with other four people.

Number of people linked to a location

MATCH (v:Victim)-[:VICTIM_OF]->(e:ViolentEvent)-[:IN_LOCATION]->(l:Location)
return l.location, count(v) as victimCount
order by victimCount desc
limit 10

╒════════════════════════════════╤═════════════╕
│"l.location"                    │"victimCount"│
╞════════════════════════════════╪═════════════╡
│"Villa Grimaldi"                │109          │
├────────────────────────────────┼─────────────┤
│"Santiago"                      │107          │
├────────────────────────────────┼─────────────┤
│"Medical Legal Institue"        │74           │
├────────────────────────────────┼─────────────┤
│"Londres No.38"                 │56           │
├────────────────────────────────┼─────────────┤
│"Cuatro Alamos"                 │55           │
├────────────────────────────────┼─────────────┤
│"Medical Legal Institute"       │49           │
├────────────────────────────────┼─────────────┤
│"San Bernardo Infantry Regiment"│49           │
├────────────────────────────────┼─────────────┤
│"Paine substation"              │36           │
├────────────────────────────────┼─────────────┤
│"La Moneda Palace"              │36           │
├────────────────────────────────┼─────────────┤
│"Chile Stadium"                 │32           │
└────────────────────────────────┴─────────────┘

Londres 38

We observe that Londres 38 appears in the list, a known place in the center of Santiago that acted as a clandestine detention center from 1973 to 1975, and is currently open to the community, acting as a memorial center (my rough translation).

There is an active group of people working on researching and exposing the activities that happened in this place, you can read more about them here (in spanish only).

From the graph, we can fetch the names of the victims that passed at some point by Londres 38.

MATCH p=(v:Victim)-[:VICTIM_OF]->(e:ViolentEvent)-[:IN_LOCATION]->(l:Location)
where l.location =~ ".*Londres.*"
return p

It is hard to represent all the names visually in one screenshot, and this is the best I could do for now.

We can get the list of names with Cypher directly

MATCH p=(v:Victim)-[:VICTIM_OF]->(e:ViolentEvent)-[:IN_LOCATION]->(l:Location)
where l.location =~ ".*Londres.*"
return l.location, v.first_name, v.last_name

╒═══════════════╤═══════════════════════╤═══════════════════════╕
│"l.location"   │"v.first_name"         │"v.last_name"          │
╞═══════════════╪═══════════════════════╪═══════════════════════╡
│"Londres No.38"│"Gregorio Antonio"     │"Gaete Farias"         │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Rodolfo Alejandro"    │"Espejo Gomez"         │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Bautista"             │"Maturana Perez"       │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Washington Hernán"    │"Maturana Perez"       │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Eduardo Enrique"      │"Lara Petrovic"        │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Enrique Segundo"      │"Toro Romero"          │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"María Elena"          │"Gonzalez Inostroza"   │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Hernán Galo"          │"Gonzalez Inostroza"   │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Alvaro Miguel"        │"Barrios Duque"        │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Muriel"               │"Dockendorff Navarrete"│
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"María Angélica"       │"Andreoli Bravo"       │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Zacarías Antonio"     │"Machuca Muñoz"        │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Juan Ernesto"         │"Ibarra Toledo"        │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Ramón Osvaldo"        │"Nuñez Espinoza"       │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"José Manuel"          │"Ramirez Rosales"      │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Sergio Daniel"        │"Tormen Mendez"        │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Luis Julio"           │"Guajardo Zamorano"    │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Juan Rosendo"         │"Chacon Olivares"      │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Alvaro Modesto"       │"Vallejos Villagrin"   │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Jorge Arturo"         │"Grez Aburto"          │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Carlos Luis"          │"Cubillos Galvez"      │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Jorque Enrique"       │"Espinoza Mendez"      │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Jaime Mauricio"       │"Buzzio Lorca"         │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Jaime del Tránsito"   │"Cadiz Norambuena"     │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Sergio Arturo"        │"Flores Ponce"         │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Luis Armando"         │"Valenzuela Figueroa"  │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Alfonso René"         │"Chanfreau Oyarce"     │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Newton Larrín"        │"Morales Saaverdra"    │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Carlos Eladio"        │"Salcedo Morales"      │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Sonia de las Mercedes"│"Bustos Reyes"         │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Joel"                 │"Huaiquiñir Benavides" │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Alejandro Arturo"     │"Parada Gonzalez"      │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Segio Sebastían"      │"Montecinos Alfaro"    │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Oscar Manuel"         │"Castro Videla"        │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Juan Aniceto"         │"Meneses Reyes"        │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Germán Rodolfo"       │"Moreno Fuenzalida"    │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Bárbara"              │"Uribe Tamblay"        │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Edwin Francisco"      │"Van Jurick Altamirano"│
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Abundio Alejandro"    │"Contreras Gonzalez"   │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Marcos Esteban"       │"Quiñones Lembach"     │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Martín"               │"Elgueta Pinto"        │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Leopoldo Daniel"      │"Muñoz Andrade"        │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Pedro Enrique"        │"Poblete Cordoba"      │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Máximo Antonio"       │"Gedda Ortiz"          │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Agustín Eduardo"      │"Reyes Gonzalez"       │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Germán Rodolfo"       │"Moreno Fuenzalida"    │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Jaime Mauricio"       │"Buzzio Lorca"         │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"José Caupolicán"      │"Villagra Astudillo"   │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Arnoldo"              │"Laurie Luengo"        │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Hernán"               │"Sarmiento Sabater"    │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Jorge Alejandro"      │"Olivares Graindorge"  │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"María Ines"           │"Alvarado Borgel"      │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Francisco Javier"     │"Fuentealba Fuentealba"│
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Artemio Segundo"      │"Gutierrez Avila"      │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"Daniel Abraham"       │"Reyes Piña"           │
├───────────────┼───────────────────────┼───────────────────────┤
│"Londres No.38"│"María Ines"           │"Alvarado Borgel"      │
└───────────────┴───────────────────────┴───────────────────────┘

From the Londres 38 site, we can learn more about some of these persons. For instance, María Ines Alvarado Borgel was taken with 21 years by national security agents (DINA). From the english version of the report (p. 690).

On July 17, 1974, DINA agents arrested the MIR activist María Inés ALVARADO BORGEL in the Providencia district. Her captors later took her to the home of Martín ELGUETA PINTO, who was also arrested along with Juan Rosendo CHACON OLIVARES. These two were active in MIR. Other persons were arrested with them but were later released. During the days after her arrest, her captors took María Inés Alvarado to her family's house several times. All three disappeared from the Londres No. 38 site, where they had been seen by witnesses. The Commission is convinced that the disappearance of these three people was the work of government agents who thus violated their human rights.

The list of the 98 persons known to date that were killed, dissapeared or died after being held captives in Londres 38 is extensively documented in https://www.londres38.cl/1937/w3-propertyvalue-35254.html. The loaded dataset is displaying 58 of them.

Wrapping up

I am stopping now. Thanks for reading until here! You can check the live repository in Github.

We can enrich this graph in many ways, but before start making up requirements, it's better to look at this data using a map. In a future post, we will expose this graph through a graphql service layer, and we will deploy a leaflet map that consumes it.

Plotting traffic maps in Barcelona

2019-08-28T23:00:00-04:00

Introduction¶

The objective is to map the traffic information for May 2019. For that we will need to get the traffic information per street section, and to connect it with the information which contains the geometry of the street sections. We can download the traffic information for last month (May, 2019), on this url

https://opendata-ajuntament.barcelona.cat/data/en/dataset/trams/resource/d335dd1a-4cd0-4eac-acb7-5ae943bcd978

We can also download the geometry of the street sections (described in terms of latitude and longitude) from this url:

https://opendata-ajuntament.barcelona.cat/data/en/dataset/transit-relacio-trams

The final goal is to have a static (e.g. an image) or interactive map, which displays the traffic state intensity by street sections, over a context background map (e.g.: OpenStreetMap).

A short disclaimer before we begin: In catalan, tram means something like segment (of a street, in this context *ahem*). It's not a rail tram.

Hands on¶

We will try to get a quick sketch using only a subset of the data first, and I will be adding comments when necessary.

We first note that the files that we will be using are really big, and we can't afford to download them from a notebook cell each time. Since this notebook was created using the Google Colab Platform, we will have to find a way to make these files available from somewhere. Luckyily, Colab's API provides a Google Drive extension that solves this problem.

In [0]:

from google.colab import drive
drive.mount('/content/drive')

Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code

Enter your authorization code:
··········
Mounted at /content/drive

In [0]:

TRAFFIC_MAY_URL="/content/drive/My Drive/Colab Notebooks/trams-barcelona-data/2019_05_Maig_TRAMS_TRAMS.csv"
STREETS_MAPPING_URL="/content/drive/My Drive/Colab Notebooks/trams-barcelona-data/transit_relacio_trams.csv"
STREETS_MAPPING_URL_LONG="/content/drive/My Drive/Colab Notebooks/trams-barcelona-data/transit_relacio_trams_format_long.csv"

In [0]:

import pandas as pd
import matplotlib.pyplot as plt

# define default size of plots
plt.rcParams["figure.figsize"] = (20,3)

trf = pd.read_csv(TRAFFIC_MAY_URL, sep=",")
trf.head(10)

Out[0]:

	idTram	data	estatActual
0	1	20190501000552	1
1	2	20190501000552	1
2	3	20190501000552	1
3	4	20190501000552	1
4	5	20190501000552	1
5	6	20190501000552	1
6	7	20190501000552	1
7	8	20190501000552	1
8	9	20190501000552	1
9	10	20190501000552	0

In [0]:

trf.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4637618 entries, 0 to 4637617
Data columns (total 4 columns):
idTram          int64
data            int64
estatActual     int64
estatPrevist    int64
dtypes: int64(4)
memory usage: 141.5 MB

In [0]:

old_size, _ = trf.shape

Some information about this dataset can be found in

https://opendata-ajuntament.barcelona.cat/data/en/dataset/trams

Here, we see that the columns EstatActual (current state) and EstatPrevist (forecasted state) have 7 possible values:

Value	Meaning
0	No data
1	Very fluid (Light)
2	Fluid
3	Dense
4	Very Dense
5	Congestion
6	Line closed

Our first task is to decide what to do with those values with 0 and 6. For the sake of these examples, we will drop these rows. In your case, you might want to work only with these rows, or something in the middle. Your call.

In [0]:

# clean trf fr
trf = trf[
    ~(
        trf['estatActual'].isin([0, 6]) | trf['estatPrevist'].isin([0, 6])
    )
]

trf.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 2257140 entries, 36 to 4637614
Data columns (total 4 columns):
idTram          int64
data            int64
estatActual     int64
estatPrevist    int64
dtypes: int64(4)
memory usage: 86.1 MB

In [0]:

new_size, _ = trf.shape

In [0]:

# print the amount of rows that were dropped
print(f"{old_size - new_size} of {old_size} rows were dropped ({(old_size-new_size)/old_size:0%})")

2380478 of 4637618 rows were dropped (51.329756%)

In [0]:

streets = pd.read_csv(STREETS_MAPPING_URL, sep=",")
streets.head(10)

Out[0]:

	Tram	Descripció	Coordenades
0	1	Diagonal (Ronda de Dalt a Doctor Marañón)	2.11203535639414,41.3841912394771,2.1015028628...
1	2	Diagonal (Doctor Marañón a Ronda de Dalt)	2.111944376806616,41.38446666680338,2.10159408...
2	3	Diagonal (Doctor Marañón a Pl. Pius XII)	2.112093343037027,41.38422850920645,2.12264979...
3	4	Diagonal (Pl. Pius XII a Doctor Marañón)	2.122592049318304,41.38719094189204,2.11196902...
4	5	Diagonal (Pl. Pius XII a Pl. Maria Cristina)	2.122657659295115,41.38694195794678,2.12755961...
5	6	Diagonal (Pl. Maria Cristina a Pl. Pius XII)	2.127445471496177,41.38841486943499,2.12260182...
6	7	Diagonal (Pl. Maria Cristina a Numància)	2.132701524574889,41.38946673844387,2.12754890...
7	8	Diagonal (Numància a Pl. Maria Cristina)	2.132462297581474,41.38967891982529,2.12746219...
8	9	Diagonal (Entença a Francesc Macià)	2.137322638833081,41.39061820620348,2.14394391...
9	10	Diagonal (Francesc Macià a Entença)	2.144060999786239,41.39265861540969,2.13711452...

In [0]:

streets.dtypes

Out[0]:

Tram            int64
Descripció     object
Coordenades    object
dtype: object

Manipulating WKT data with shapely¶

In [0]:

streets["Coordenades"].iloc[1]

Out[0]:

'2.111944376806616,41.38446666680338,2.101594089443895,41.38186790291103'

Here we note that the coordinates are in chunks of text and there's no encoding explicited (I didn't find any on the official information). We need to do some research.

But before diving in, let's check for missing data.

In [0]:

# we check for possible missing data
streets["Coordenades"].isna().sum()

Out[0]:

In [0]:

streets["Coordenades"].isnull().sum()

Out[0]:

In [0]:

expanded_coords = streets["Coordenades"].str.split(",", expand=True)
expanded_coords.head()

Out[0]:

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32	33	34	35	36	37	38	39	40	41	42	43	44	45	46	47	48	49	50	51	52	53	54	55	56	57	58	59	60	61	62	63	64	65	66	67	68	69	70	71	72	73	74	75
0	2.11203535639414	41.3841912394771	2.101502862881051	41.3816307921222	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None
1	2.111944376806616	41.38446666680338	2.101594089443895	41.38186790291103	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None
2	2.112093343037027	41.38422850920645	2.122649794166167	41.38692960796057	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None
3	2.122592049318304	41.38719094189204	2.111969021588291	41.38445704285778	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None
4	2.122657659295115	41.38694195794678	2.127559612359478	41.38817909013056	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None	None

In [0]:

len(expanded_coords.columns)

Out[0]:

We note that the Coordenades column could be expanded into ~76 new columns, if we consider sep="," as a separator. From looking at this data, it's probably in the WKT format (Well-Known Text representation of geometry) and it represents a line, according to the LINESTRING representation.

We'll need to convert each row to match this WKT format, using string manipulation. But we don't want to reinvent the wheel, so we will be using a python package called Shapely developed specifically for these types of tasks.

In [0]:

!pip install shapely

Requirement already satisfied: shapely in /usr/local/lib/python3.6/dist-packages (1.6.4.post2)

In [0]:

sample = streets["Coordenades"].str.split(",")[0]
sample

Out[0]:

['2.11203535639414',
 '41.3841912394771',
 '2.101502862881051',
 '41.3816307921222']

Reshaping the strings¶

A LINESTRING is defined in terms of pairs of values. We need to take a list of length $n$, (and assume this number is even), and reshape it as a list of tuples.

[a, b, c, d, e, f, g] -> [(a, b), (c, d), (f, g),]

In [0]:

import typing as T

from itertools import zip_longest

# not tested
def reshape_line(line: T.List[str], chunksize: int, fillvalue: int) -> T.Iterable[T.Tuple[float]]:
    """Reshape a line to match the LineString WKT format
    
    This is based on `zip_longest`, read more in 
    <https://docs.python.org/3/library/itertools.html#itertools.zip_longest> and
    in the StackOverflow solution posted in <https://stackoverflow.com/a/434411/5819113>
    
    
        zip_longest('ABCD', 'xy', fillvalue='-') --> Ax By C- D-
       
    """
    args = [iter(float(el) for el in line)] * chunksize
    return zip_longest(*args, fillvalue=fillvalue)

In [0]:

list(reshape_line(sample, chunksize=2, fillvalue=0))

Out[0]:

[(2.11203535639414, 41.3841912394771), (2.101502862881051, 41.3816307921222)]

With this, we can get our very first drawing. Baby steps.

In [0]:

from shapely.geometry import LineString

sample_wkt = LineString(reshape_line(sample, 2, 0))
sample_wkt

Out[0]:

The next step is to apply this process to every row on the column Coordenades. We will implement a helper function for this purpose.

In [0]:

def convert_line_to_wkt(line: str) -> LineString:
    splitted_line = line.split(",")
    reshaped_line = reshape_line(splitted_line, chunksize=2, fillvalue=None)
    return LineString(reshaped_line)

In [0]:

# let's process the whole column
coords_wkt = streets["Coordenades"].apply(lambda line: convert_line_to_wkt(line))

# access the __str__ of the first element
print(coords_wkt[1])

LINESTRING (2.111944376806616 41.38446666680338, 2.101594089443895 41.38186790291103)

With the values in WKT format, we could use an online WKT viewer like http://arthur-e.github.io/Wicket/sandbox-gmaps3.html to see how this is rendered on a map.

First plot using `geopandas`¶

The former was the first approach. We will try now our next step: plot everything at once. Whatever gets displayed will be considered success.

In [0]:

!pip install geopandas

Collecting geopandas
  Downloading https://files.pythonhosted.org/packages/21/80/da2a33c9201cd4ce693f4aa6189efc9ef1a48bec1c3b02c3ce9908b07fec/geopandas-0.5.1-py2.py3-none-any.whl (893kB)
     |████████████████████████████████| 901kB 5.1MB/s 
Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from geopandas) (0.24.2)
Requirement already satisfied: shapely in /usr/local/lib/python3.6/dist-packages (from geopandas) (1.6.4.post2)
Collecting fiona (from geopandas)
  Downloading https://files.pythonhosted.org/packages/89/4a/193cd6a75e51062c85f4e1cd6f312b3bbda6e26ba7510f152ef5016f0b16/Fiona-1.8.6-cp36-cp36m-manylinux1_x86_64.whl (17.9MB)
     |████████████████████████████████| 17.9MB 41.2MB/s 
Collecting pyproj (from geopandas)
  Downloading https://files.pythonhosted.org/packages/16/59/43869adef45ce4f1cf7d5c3aef1ea5d65d449050abdda5de7a2465c5729d/pyproj-2.2.1-cp36-cp36m-manylinux1_x86_64.whl (11.2MB)
     |████████████████████████████████| 11.2MB 22.2MB/s 
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas->geopandas) (2018.9)
Requirement already satisfied: numpy>=1.12.0 in /usr/local/lib/python3.6/dist-packages (from pandas->geopandas) (1.16.4)
Requirement already satisfied: python-dateutil>=2.5.0 in /usr/local/lib/python3.6/dist-packages (from pandas->geopandas) (2.5.3)
Requirement already satisfied: click<8,>=4.0 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas) (7.0)
Collecting click-plugins>=1.0 (from fiona->geopandas)
  Downloading https://files.pythonhosted.org/packages/e9/da/824b92d9942f4e472702488857914bdd50f73021efea15b4cad9aca8ecef/click_plugins-1.1.1-py2.py3-none-any.whl
Requirement already satisfied: attrs>=17 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas) (19.1.0)
Collecting munch (from fiona->geopandas)
  Downloading https://files.pythonhosted.org/packages/68/f4/260ec98ea840757a0da09e0ed8135333d59b8dfebe9752a365b04857660a/munch-2.3.2.tar.gz
Requirement already satisfied: six>=1.7 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas) (1.12.0)
Collecting cligj>=0.5 (from fiona->geopandas)
  Downloading https://files.pythonhosted.org/packages/e4/be/30a58b4b0733850280d01f8bd132591b4668ed5c7046761098d665ac2174/cligj-0.5.0-py3-none-any.whl
Building wheels for collected packages: munch
  Building wheel for munch (setup.py) ... done
  Created wheel for munch: filename=munch-2.3.2-py2.py3-none-any.whl size=6613 sha256=5157e6f60cca69d1d04d0bb4cc14e5d53b0702ddfe97d8af306c747f4137671d
  Stored in directory: /root/.cache/pip/wheels/db/bf/bc/06a3e1bfe0ab27d2e720ceb3cff3159398d92644c0cec2c125
Successfully built munch
Installing collected packages: click-plugins, munch, cligj, fiona, pyproj, geopandas
Successfully installed click-plugins-1.1.1 cligj-0.5.0 fiona-1.8.6 geopandas-0.5.1 munch-2.3.2 pyproj-2.2.1

In [0]:

from geopandas import GeoDataFrame

crs = {'init': 'epsg:4326'}
gdf = GeoDataFrame(streets, crs=crs, geometry=coords_wkt)
ax = gdf.plot(cmap="viridis")

Note that here, the colormap viridis is not supposed to make sense. It is not giving any useful information out of the box. But we'll get to that in a minute.

Tweaking the plot¶

Hooray! We got a map. But it's ugly and it doesn't say much.

At this point we could start tweaking the plot using the matplotlib API. We could add legends, meaningful colours, different line styles, etc. You get it. And if you have played with matplotlib long enough, you'll know that almost everything is possible, but not easy to get.

In our case we don't need to produce beautiful plots at this point, we need to provide an efficient way to explore the data visually.

For maps, the best tool to do this IMHO is to use leaflet, but we don't want to write javascript (I don't want to). We will be using Folium, which is a library for using leaflet in python.

Plotting routes on top of Barcelona using `Folium`¶

In [0]:

!pip install folium

Requirement already satisfied: folium in /usr/local/lib/python3.6/dist-packages (0.8.3)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from folium) (1.16.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.6/dist-packages (from folium) (2.10.1)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from folium) (1.12.0)
Requirement already satisfied: branca>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from folium) (0.3.1)
Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from folium) (2.21.0)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.6/dist-packages (from jinja2->folium) (1.1.1)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->folium) (1.24.3)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->folium) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->folium) (2019.6.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->folium) (3.0.4)

In [0]:

import os
import folium

print(folium.__version__)

0.8.3

In [0]:

barcelona_map = folium.Map([41.3947,2.1557], zoom_start=12.4, tiles='cartodbpositron')
folium.GeoJson(gdf).add_to(barcelona_map)
# plot map
barcelona_map

Out[0]:

Plotting traffic data¶

This map is plain and boring, because it doesn't contain any information from the traffic dataset. Let's fix that.

We would like to achieve the following:

To draw the lines from Coord_wkt on top of a map of Barcelona
To plot the values from the trf dataframe, showing how busy is the line at a certain moment, defined by the column data

We note that the traffic data has a temporal component in data, which will benefit greatly from using the correct datatype, i.e. we will try to use the DateTime type over the Object type.

In [0]:

trf["data"].head()

Out[0]:

36     20190501000552
56     20190501000552
57     20190501000552
143    20190501000552
276    20190501000552
Name: data, dtype: int64

We can convert this column to datetime (https://pandas.pydata.org/pandas-docs/version/0.20/generated/pandas.to_datetime.html) following the strftime format specifications:

In [0]:

# https://pandas.pydata.org/pandas-docs/version/0.20/generated/pandas.to_datetime.html
pd.to_datetime(trf["data"].astype(str)[:5], format='%Y%m%d%H%M%S')

Out[0]:

36    2019-05-01 00:05:52
56    2019-05-01 00:05:52
57    2019-05-01 00:05:52
143   2019-05-01 00:05:52
276   2019-05-01 00:05:52
Name: data, dtype: datetime64[ns]

In [0]:

trf["data_datetime"] = pd.to_datetime(trf["data"].astype(str), format='%Y%m%d%H%M%S')

In [0]:

trf.columns

Out[0]:

Index(['idTram', 'data', 'estatActual', 'estatPrevist', 'data_datetime'], dtype='object')

In [0]:

trf.head(5)

Out[0]:

	idTram	data	estatActual	estatPrevist	data_datetime
36	37	20190501000552	4	5	2019-05-01 00:05:52
56	57	20190501000552	1	5	2019-05-01 00:05:52
57	58	20190501000552	1	5	2019-05-01 00:05:52
143	144	20190501000552	1	5	2019-05-01 00:05:52
276	277	20190501000552	2	5	2019-05-01 00:05:52

Quick survey on the data¶

How busy are trams compared to the expected traffic, on average, for the month of May?¶

As an extra, we'll try to assess the forecasts: were they right? We'll consider a simple rule of thumb

If the forecasted traffic is greater than the current traffic, then the forecast is right

This is one approach, and I agree it is rough. There are multiple ways of asessing the error in predictions, but here's not the place to discuss them.

In [0]:

# caveat: this is an *aggregated* measure of the whole month, but it provides
# a rough measure

monthly_traffic = trf.groupby(by=['idTram'], as_index=True).mean()
monthly_traffic["diff_perc"] = monthly_traffic["estatActual"]/monthly_traffic["estatPrevist"]*100
monthly_traffic["was_forecast_right"] = monthly_traffic["diff_perc"] <= 100
monthly_traffic.head()

Out[0]:

	data	estatActual	estatPrevist	diff_perc	was_forecast_right
idTram
1	2.019052e+13	2.127508	2.125576	100.090890	False
2	2.019052e+13	1.809214	1.874505	96.516888	True
3	2.019052e+13	1.985931	2.011511	98.728342	True
4	2.019052e+13	2.262626	2.305628	98.134936	True
5	2.019052e+13	2.744457	2.737290	100.261860	False

How busy were trams on average, on an hour basis, per tram line, during the month of May?¶

In [0]:

# line continuation according to PEP8 
# see https://stackoverflow.com/a/17630918/5819113

trams_hourly = (
    # drop unwanted columns
    trf.drop(columns=['data', 'idTram'])
        # group by desired columns
        .groupby(by=[trf['idTram'], trf["data_datetime"].dt.day], as_index=True)
        # show the mean of the aggregated values
        .mean()
)

trams_hourly["diff_perc"] = trams_hourly["estatActual"]/trams_hourly["estatPrevist"]*100
trams_hourly["was_forecast_right"] = trams_hourly["diff_perc"] <= 100
trams_hourly.head(24) # first 24 hours

Out[0]:

		estatActual	estatPrevist	diff_perc	was_forecast_right
idTram	data_datetime
1	1	1.598131	1.696262	94.214876	True
	2	2.178082	2.091324	104.148472	False
	3	2.452915	2.313901	106.007752	False
	4	1.771300	1.793722	98.750000	True
	5	1.648649	1.675676	98.387097	True
	6	2.256881	2.215596	101.863354	False
	7	2.261538	2.230769	101.379310	False
	8	2.278302	2.264151	100.625000	False
	9	2.255814	2.241860	100.622407	False
	10	2.451613	2.479263	98.884758	True
	11	1.724138	1.719828	100.250627	False
	12	1.654867	1.681416	98.421053	True
	13	2.253333	2.235556	100.795229	False
	14	2.197183	2.169014	101.298701	False
	15	2.459016	2.420765	101.580135	False
	16	2.386935	2.351759	101.495726	False
	17	2.384615	2.380090	100.190114	False
	18	1.758772	1.771930	99.257426	True
	19	1.583710	1.628959	97.222222	True
	20	2.326829	2.321951	100.210084	False
	21	2.160194	2.223301	97.161572	True
	22	2.179487	2.215385	98.379630	True
	23	2.385650	2.426009	98.336414	True
	24	2.563319	2.502183	102.443281	False

Baking everything together¶

Let's merge dataframes and see what we get. This is my first time using these libraries, so my code has a lot of comments that hopefully explain what I'm doing. I've taken a lot from the examples given in the folium repo, so please go there to see what else can be done!

In [0]:

# join dataframes, to *see* what happens. But make a copy first.
_trf_to_merge = trf.copy()
_streets_to_merge = streets.copy()

In [0]:

# merge both dataframes using the id of the tram as common column
_merged = _trf_to_merge.merge(_streets_to_merge, left_on="idTram", right_on="Tram")
# drop unnecesary columns and set the idTram
_merged = _merged.drop(columns=["data", "Tram", "Coordenades"])
_merged.head()

Out[0]:

	idTram	estatActual	estatPrevist	data_datetime	Descripció	geometry
0	37	4	5	2019-05-01 00:05:52	Creu Coberta - Sants (Gran Via Carles III a Ri...	LINESTRING (2.121871003054767 41.3757077900903...
1	37	3	5	2019-05-01 00:00:53	Creu Coberta - Sants (Gran Via Carles III a Ri...	LINESTRING (2.121871003054767 41.3757077900903...
2	37	3	5	2019-05-01 00:10:53	Creu Coberta - Sants (Gran Via Carles III a Ri...	LINESTRING (2.121871003054767 41.3757077900903...
3	37	2	3	2019-05-01 00:15:52	Creu Coberta - Sants (Gran Via Carles III a Ri...	LINESTRING (2.121871003054767 41.3757077900903...
4	37	3	3	2019-05-01 00:20:52	Creu Coberta - Sants (Gran Via Carles III a Ri...	LINESTRING (2.121871003054767 41.3757077900903...

In [0]:

# `estatActual` and `estatPrevist` vary from 1 to 5
_merged.describe()

Out[0]:

	idTram	estatActual	estatPrevist
count	2.125642e+06	2.125642e+06	2.125642e+06
mean	2.520718e+02	1.896606e+00	1.925726e+00
std	1.606364e+02	8.496675e-01	9.009781e-01
min	1.000000e+00	1.000000e+00	1.000000e+00
25%	1.170000e+02	1.000000e+00	1.000000e+00
50%	2.240000e+02	2.000000e+00	2.000000e+00
75%	4.080000e+02	2.000000e+00	2.000000e+00
max	5.340000e+02	5.000000e+00	5.000000e+00

In [0]:

_merged.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 2125642 entries, 0 to 2125641
Data columns (total 6 columns):
idTram           int64
estatActual      int64
estatPrevist     int64
data_datetime    datetime64[ns]
Descripció       object
geometry         object
dtypes: datetime64[ns](1), int64(3), object(2)
memory usage: 113.5+ MB

Plotting combined data¶

Grouping information by day¶

In [0]:

from geopandas import GeoDataFrame

crs = {'init': 'epsg:4326'}
merged_gdf = GeoDataFrame(_merged, crs=crs, geometry=_merged['geometry'])

In [0]:

# create a colormap using branca
# following the examples from
# https://nbviewer.jupyter.org/github/python-visualization/folium/blob/master/examples/plugin-Search.ipynb

import branca

colorscale = branca.colormap.linear.viridis.scale(1, 5)
colorscale.caption="Tram usage level"


def style_function(feature):
    estat_actual = int(feature['properties']['estatActual']) or None
    return {
        'alpha': 0.5,
        'weight': 3,
        'color': '#black' if estat_actual is None else colorscale(estat_actual)
    }

In [0]:

# we will manipulate the dataframe to plot data using time scales, 
# since plotting the whole dataframe it's too expensive memory-wise

first_10_am = merged_gdf[(merged_gdf["data_datetime"].dt.day == 1) & (merged_gdf["data_datetime"].dt.hour == 10)]

In [0]:

# we need to drop this column because it raises an error
first_10_am = first_10_am.drop(columns="data_datetime", inplace=False)

In [0]:

# plot the map
barca_map_colors = folium.Map([41.3947,2.1557], zoom_start=12.4, tiles='cartodbpositron')

(
    folium.GeoJson(
        first_10_am,
        style_function=style_function)
    .add_to(barca_map_colors)
)

# add legend, see <https://github.com/python-visualization/folium/issues/528>
barca_map_colors.add_child(colorscale)

# plot map
barca_map_colors

Out[0]:

In [0]:

# we will manipulate the dataframe to plot data using time scales

gdf_hourly = (
        # group by desired columns
        merged_gdf.groupby(
            by=[merged_gdf['idTram'],
                merged_gdf["data_datetime"].dt.day, # here is where the grouping happens
               ],
            as_index=True)
        # show the mean of the aggregated values
        .mean()
        .reset_index()
)

gdf_hourly["diff_perc"] = gdf_hourly["estatActual"]/gdf_hourly["estatPrevist"]*100
gdf_hourly["was_forecast_right"] = gdf_hourly["diff_perc"] <= 100
gdf_hourly_w_geometry = gdf_hourly.merge(streets, left_on="idTram", right_on="Tram")
gdf_hourly_w_geometry.tail()

Out[0]:

	idTram	data_datetime	estatActual	estatPrevist	diff_perc	was_forecast_right	Tram	Descripció	Coordenades	geometry
10281	534	27	1.883392	1.886926	99.812734	True	534	Ronda Litoral (Nus de la Trinitat a Potosí)	2.206062333803802,41.44356616311283,2.20196194...	LINESTRING (2.206062333803802 41.4435661631128...
10282	534	28	1.981413	2.144981	92.374350	True	534	Ronda Litoral (Nus de la Trinitat a Potosí)	2.206062333803802,41.44356616311283,2.20196194...	LINESTRING (2.206062333803802 41.4435661631128...
10283	534	29	1.929577	1.978873	97.508897	True	534	Ronda Litoral (Nus de la Trinitat a Potosí)	2.206062333803802,41.44356616311283,2.20196194...	LINESTRING (2.206062333803802 41.4435661631128...
10284	534	30	1.851064	1.918440	96.487985	True	534	Ronda Litoral (Nus de la Trinitat a Potosí)	2.206062333803802,41.44356616311283,2.20196194...	LINESTRING (2.206062333803802 41.4435661631128...
10285	534	31	2.017544	2.056140	98.122867	True	534	Ronda Litoral (Nus de la Trinitat a Potosí)	2.206062333803802,41.44356616311283,2.20196194...	LINESTRING (2.206062333803802 41.4435661631128...

Plot daily average values¶

We are using a discrete colorscale that ranges from 1 to 6, to indicate the mean of the estatActual for each Tram line on May 27 (The color selection is arbitrary)
We are drawing the lines whose capacity forecast was right (e.g. the actual capacity was below the forecasted value) with a dashed line, and those lines whose value was wrong (e.g. the capacity forecast fell short on that day) with a continuous line. This is because of the limited styles that leaflet provides out of the box, and any work on improving this style had to be done with CSS, which is out of the scope of this challenge.

In [0]:

import branca
from folium.plugins import Search
from geopandas import GeoDataFrame

# produce the geopandas dataframe
crs = {'init': 'epsg:4326'}
gdf_hourly = GeoDataFrame(
    gdf_hourly_w_geometry[gdf_hourly_w_geometry['data_datetime']==27], # here is where the filtering happens
    crs=crs,
    geometry=gdf_hourly_w_geometry['geometry']
)

# create colorscale
colorscale = branca.colormap.step.Accent_06.scale(1, 5)
colorscale.caption="estatActual (Daily average)"

# define a method for styling lines
def style_function_capacity(feature):
    not_exceeded = int(feature['properties']['was_forecast_right']) or None
    estat_actual = int(feature['properties']['estatActual']) or None
    
    # see for other options https://leafletjs.com/reference-1.5.0.html#path-option
    return {
        'alpha': 0.5,
        'dashArray': '5,10' if not_exceeded else '0', # dashed line unless forecast is wrong
        'weight': 3,
        'color': '#black' if estat_actual is None else colorscale(estat_actual),
    }
  
# plot the map of barcelona
hourly_map = folium.Map([41.3947,2.1557], zoom_start=12.4, tiles='cartodbpositron')

# plot the tram lines
tram_map = (
    folium.GeoJson(
        gdf_hourly,
        name="Barcelona Tram Capacity (27 May 2019)",
        style_function=style_function_capacity,
        tooltip=folium.GeoJsonTooltip(
            fields=['idTram', 'estatActual', 'estatPrevist', 'diff_perc', 'Descripció'],
            aliases=['Tram ID', 'Observed Capacity', 'Forecasted Capacity', 'Load (%)', 'Description'], 
            localize=True)
    )
    .add_to(hourly_map)
)

# add legend, see <https://github.com/python-visualization/folium/issues/528>
hourly_map.add_child(colorscale)

# add a searchbox bound to the Description of lines
statesearch = Search(
    layer=tram_map,
    geom_type='Line',
    placeholder='Search for a Tram Description',
    collapsed=False,
    search_label='Descripció',
    weight=3
).add_to(hourly_map)


# plot map
hourly_map

Out[0]:

Final words¶

Thanks for reading until here! Regarding the data, I've explored manually isolated cases, so my conclusions need a bit of work. I'll refer only to the last map being shown (May 27).

It's not common to observe capacity levels over 3
Traffic loads range from 90 to 110%. Forecasts are really tight but it seems there's no need for accounting for more.
It may be worth exploring plots of diff_perc and other error measurements.

It's easy to extend these maps to any time of the day, using the dt accessor and using the data_datetime as a pivot for indexing.

My intention with this notebook is to show the power of leaflet and python, and the libraries available to manipulate geocoded data like geopandas and shapely. I've found that there is an application that uses this data, check https://com-shi-va.barcelona.cat/en/transit-

Until next time.

Dumping postgres databases with Docker

2019-03-14T21:10:00+01:00

It's been a while since I posted something. I started living in another country, and time has become scarce. This is a short post from my developer journal.

Docker and postgres work really good together. If you don't believe me, check their image documentation.

Take for instance, the barebones example given there:

# Use postgres/example user/password credentials
version: '3.1'

services:

  db:
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: example

  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080

With this file, you can get a living postgres instance in seconds with something like docker-compose up -d.

Managing database backups with `pg_dump`

The simplest way to backup a postgres database is using pg_dump. From the host, you can issue this using

docker-compose exec db pg_dump -U postgres postgres --no-owner | gzip -9  > db-backup-$(date +%d-%m-%y).sql.gz

This executes pg_dump inside the running container specified in the docker-compose.yml file as the service db, and stores the result in the host as a gzip compressed file.

Here, passing --no-owner ensures that the database backup will be importable into any other database, regardless of user privileges. Use this with care.

Importing this backup into another database living in another container can be done in two ways:

copying the file or mounting it in the container and importing it, from inside the container
Pipe the file from the host to the container through stdin

I personally like the second approach better, as it doesn't require modifying any files inside the container:

gunzip -c path/to/db-backup.sql.gz | docker-compose exec -T db psql -U postgres postgres

Note that we need to pass -T to disable TTY allocation. Also, we could use docker instead of docker-compose, with something like docker exec -i $(docker-compose ps -q db).

These tasks can scheduled to be run at specific times with cron. More info on backups here.

Using psycopg2 for your postgres needs

2018-12-03T23:00:00-03:00

1 About
2 Postgres Mission
- 2.1 Create the postgres instance
3 Create users and roles
4 Preparing for data insertion
5 Grant specific privileges between users and tables
- 5.1 Verify permissions and roles
6 Final words
7 Moving forward

About¶

This is the notebook I created to solve the first mission in the data engineering path in dataquest.io. I am also experimenting with the Pelican-Jupyter Notebook extension, that allows to load jupyter notebooks directly in the blog.

Postgres Mission¶

The problem goes something like this:

Recently, the International Hurricane Watchgroup (IHW) has been asked to update their analysis tools. Because of the increase in public awareness of hurricanes, they are required to be more diligient with the analysis of historical hurricane data they share across the organization. They have asked you, someone with experience in databases, to help work with their team to productionize their services.

Accepting the job, their team tells you that they have been having trouble sharing data across the teams and keeping it consistent. From what they've told you, it seems that their method of sharing the data with their data anaylsts has been to save a CSV file on their local servers and have every data analyst pull the data down. Then, each analyst uses a local SQLite engine to store the CSV, run their queries, and send their results around.

From what they have told you, you might be thinking that this is an inefficient way of sharing data. To understand what you will be working on, they have sent you a CSV file. Their CSV file contains the following fields:
fid - ID for the row
year - Recorded year
month - Recorded month
day - Recorded date
ad_time - Recorded time in UTC
btid - Hurricane ID
name - Name of the hurricane
lat - Latitude of the recorded location
long - Longitude of the recorded location
wind_kts - Wind speed in knots per second
pressure - Atmospheric pressure of the hurricane
cat - Hurricane category
basin - The basin the hurricane is located
shape_leng - Hurricane shape length
In this Guided Project, you will be using the local installed version of Postgres you installed from the previous project. This is much different than previous Guided Projects as you will be using your own notebook and your own Python environment instead of Dataquest's. Your job is to show that you can create a database that will accomplish the following requirements:

Database for the IHW to store their tables.

Table that contains the fields detailed in the CSV file

User that can update, read, and insert into a table of the data.

Insert the data into the table.

Let's do this.

Create the postgres instance¶

We'll be using a docker container with a postgres persistent database. The pgadmin instance is not needed, but we can use it to have a GUI available by default in 0.0.0.0:8080. Note this is a toy configuration, and it should not be used to store sensitive information.

We won't be dealing with what this file means. Here we use docker to have a disposable database quickly. You could have a local postgres instance and that would work as well.

You can get more information about postgres in docker in the docker store

In [21]:

!cat docker-compose.yml

# Use postgres/example user/password credentials
version: '3.1'

services:
  db:
    image: postgres:10
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres 
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - 54320:5432
    networks:
      db_nw:
        aliases:
          - postgres
  pgadmin:
    image: dpage/pgadmin4:latest
    environment:
      POSTGRES_PASSWORD: postgres 
      PGADMIN_DEFAULT_EMAIL: user@domain.com 
      PGADMIN_DEFAULT_PASSWORD: admin
    ports:
      - 8080:80
    networks:
      - db_nw

volumes:
    pgdata: {}

networks:
  db_nw:
    driver: bridge

In [22]:

!docker-compose -p dq_de up -d

dq_de_db_1 is up-to-date
dq_de_pgadmin_1 is up-to-date

In [23]:

# check if postgres docker instance is alive
!docker ps

CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS              PORTS                           NAMES
58b58e1f97f4        postgres:10                             "docker-entrypoint.s…"   10 days ago         Up 3 hours          0.0.0.0:54320->5432/tcp         dq_de_db_1
c5b7c0ec8430        dpage/pgadmin4:latest                   "/entrypoint.sh"         10 days ago         Up 3 hours          443/tcp, 0.0.0.0:8080->80/tcp   dq_de_pgadmin_1
9f18df162b11        rediscommander/redis-commander:latest   "/usr/bin/dumb-init …"   2 weeks ago         Up 3 hours          0.0.0.0:8081->8081/tcp          redis-commander
e38a793b3b60        postgres:10                             "docker-entrypoint.s…"   2 weeks ago         Up 3 hours          0.0.0.0:51085->5432/tcp         demo_db_1

In [29]:

# retrieve the ip address of the container, seen from the host
!docker inspect dq_de_db_1 | grep "IPAddress"

            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "192.168.32.3",

Note that we could also refer to our database by the port mapping done in the docker-compose.yml file. This means that we could reach our database in 0.0.0.0:54320.

Create users and roles¶

The mission instructions state that

With a table set up, it's now time to create a user on the Postgres database that can insert, update, and read the data but not delete. This is to make sure that someone who might get a hold of this user does not issue a destructive command. Essentially, this is like creating a "data production" user whose job it is is to always write new and existing data to the table.

Futhermore, even though it wasn't according to the spec, we know that the IHW team's analysts just run read queries on the data. Also, since the analysts only know SQLite queries, they may not be well-versed in a production database. As such, it might be risky handing out a general production user for them to query their data.

From what you have learned about security and restricting well meaning users, it might be a good idea to restrict those analysts from some commands. Those commands can be anything from adding new data to the table or changing the values. You should decide what commands should be given to the analyst user.

In short, what I get from this is

Connected as a postgres user, create a user data_prod_user that can insert, update, and read the data but not delete.
This is like a "data production" user whose job is to always write new and existing data to the table.

Also,

Create a role GROUP for the IHW analyst team that can only read data from tables
Create a user in this group, called analyst1. Picking names is hard

Note that users and groups are all part of the same entity in postgres, the role.

In [1]:

import psycopg2

conn = psycopg2.connect(
    dbname="postgres", user="postgres", password="postgres", host="192.168.32.3"
)

conn.autocommit = True
cur = conn.cursor()

DATA_PROD_USER_PWD = "changeme"
DATA_VIEW_USER_PWD = "changeme"

# create user and detach password from command
# also add an IF NOT EXIST clause as a hack from
# https://stackoverflow.com/questions/8092086/create-postgresql-role-user-if-it-doesnt-exist
cur.execute(f"""
CREATE USER data_prod_user WITH PASSWORD '{DATA_PROD_USER_PWD}';
CREATE GROUP readonly WITH NOLOGIN;
CREATE USER analyst1 WITH PASSWORD '{DATA_VIEW_USER_PWD}' IN GROUP readonly;
""")

# create db
print("CREATE DATABASE ihw OWNER postgres")
cur.execute("CREATE DATABASE ihw OWNER postgres")

conn.commit()
conn.close()

/home/diego/.pyenv/versions/3.6.6/envs/dq-de/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)

CREATE DATABASE ihw OWNER postgres

If you want to start over, just DROP everything.

In [26]:

# revoke all permissions by default. 
# Use this to start all over.
import psycopg2

conn = psycopg2.connect(
    dbname="postgres", user="postgres", password="postgres", host="192.168.32.3"
)

conn.autocommit = True
cur = conn.cursor()

# drop users
cur.execute("""
DROP USER data_prod_user;
DROP USER analyst1;
DROP GROUP readonly;
""")

# drop db
print("DROP DATABASE IF EXISTS ihw")
cur.execute("DROP DATABASE IF EXISTS ihw")

conn.commit()
conn.close()

Preparing for data insertion¶

Now with our new users, we can start creating our tables. A table is assigned by default to the user that created it, and it will live by default in the database that the connection was assigned to.

The steps involved in inserting the data are

Inspect the data: This process is specific to the problem, and it may vary each time. I will be using pandas
Infer the data types from the previous analysis that will be used in out SQL table: This refers to the questions of the type "would an integer suffice?", "how long should these strings be", "Can this column be normalized?". This will probably require shaping the data analyzed previously.
Create the table: This is the translation of the previous step into a proper SQL table.
Populate the table: Fill the new table with data. I will be using postgres' COPY feature by means of psycopg.copy_expert()

Inspection of the data¶

Before even knowing how our table will should be created, we should inspect our data. We will be using pandas for this purpose.

In [1]:

# load data
import pandas as pd
df = pd.read_csv("https://dq-content.s3.amazonaws.com/251/storm_data.csv")
df.head()

Out[1]:

	FID	YEAR	MONTH	DAY	AD_TIME	BTID	NAME	LAT	LONG	WIND_KTS	CAT	BASIN	Shape_Leng
0	2001	1957	8	8	1800Z	63	NOTNAMED	22.5	-140.0	50	TS	Eastern Pacific	1.140175
1	2002	1961	10	3	1200Z	116	PAULINE	22.1	-140.2	45	TS	Eastern Pacific	1.166190
2	2003	1962	8	29	0600Z	124	C	18.0	-140.0	45	TS	Eastern Pacific	2.102380
3	2004	1967	7	14	0600Z	168	DENISE	16.6	-139.5	45	TS	Eastern Pacific	2.121320
4	2005	1972	8	16	1200Z	251	DIANA	18.5	-139.8	70	H1	Eastern Pacific	1.702939

In [2]:

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 59228 entries, 0 to 59227
Data columns (total 14 columns):
FID           59228 non-null int64
YEAR          59228 non-null int64
MONTH         59228 non-null int64
DAY           59228 non-null int64
AD_TIME       59228 non-null object
BTID          59228 non-null int64
NAME          59228 non-null object
LAT           59228 non-null float64
LONG          59228 non-null float64
WIND_KTS      59228 non-null int64
PRESSURE      59228 non-null int64
CAT           59228 non-null object
BASIN         59228 non-null object
Shape_Leng    59228 non-null float64
dtypes: float64(3), int64(7), object(4)
memory usage: 6.3+ MB

In [3]:

df.describe()

Out[3]:

	FID	YEAR	MONTH	DAY	BTID	LAT	LONG	WIND_KTS	PRESSURE	Shape_Leng
count	59228.000000	59228.000000	59228.000000	59228.000000	59228.000000	59228.000000	59228.000000	59228.000000	59228.000000	59228.000000
mean	29614.500000	1957.194874	8.540521	15.867326	648.398899	23.526400	-83.196863	54.726802	372.336800	1.201987
std	17097.795209	41.665792	1.364174	8.793432	372.376803	9.464955	37.282152	25.133577	480.562974	0.834497
min	1.000000	1851.000000	1.000000	1.000000	1.000000	4.200000	-180.000000	10.000000	0.000000	0.000000
25%	14807.750000	1928.000000	8.000000	8.000000	344.000000	16.100000	-108.500000	35.000000	0.000000	0.707107
50%	29614.500000	1970.000000	9.000000	16.000000	606.000000	21.200000	-81.200000	50.000000	0.000000	1.029563
75%	44421.250000	1991.000000	9.000000	23.000000	920.000000	29.600000	-62.200000	70.000000	990.000000	1.431782
max	59228.000000	2008.000000	12.000000	31.000000	1410.000000	69.000000	180.000000	165.000000	1024.000000	11.180340

Fortunately, we have some guidelines from dataquest of how the table should look like:

fid is an INT and should be a PRIMARY KEY
YEAR, MONTH, DAY and AD_TIME should be merged into one single TIMESTAMP, using TIME ZONE.
btid looks like an INT
We have no way of knowing how long a name can be, thus we will be using a VARCHAR
lat is of type DECIMAL(X,Y). We need to figure out X and Y
long is of type DECIMAL(X,Y), We also need to figure out X and Y
wind_kts looks like an INT
pressure looks like an INT
cat looks like a categorical value, kinda like a VARCHAR(X). We need to figure out X
basin is a VARCHAR(X). Same as cat
shape_length looks like a DECIMAL(X, Y)

The next step is figuring out some variables, like

The values of X and Y in VARCHAR and DECIMAL
The maximum values of those columns that look like integers. With this we could use even smaller datatypes like SMALLINT and others.

We can infer most of these values through inspection of the min and max values in df.describe(). We could then produce a table like this:

CREATE TABLE hurricanes (
    fid INT PRIMARY KEY,
    timestamp TIMESTAMP WITH TIME ZONE,
    btid SMALLINT,
    name VARCHAR,
    lat DECIMAL(8,6),
    long DECIMAL(9,6),
    wind_kts SMALLINT,
    pressure SMALLINT,
    cat VARCHAR(X),
    basin VARCHAR(X),
    shape_length DECIMAL(8, 6)
    )

We can infer the values of X in VARCHAR(X) by doing something like

In [6]:

df['BASIN'].str.len().max()

Out[6]:

In [7]:

df['CAT'].str.len().max()

Out[7]:

This means that these columns are maximum 15 and 2 characters long. Out table will end looking like

CREATE TABLE hurricanes (
    fid INT PRIMARY KEY,
    timestamp TIMESTAMP WITH TIME ZONE,
    btid SMALLINT,
    name VARCHAR,
    lat DECIMAL(8,6),
    long DECIMAL(9,6),
    wind_kts SMALLINT,
    pressure SMALLINT,
    cat VARCHAR(2),
    basin VARCHAR(15),
    shape_length DECIMAL(8, 6)
    )

Shaping data¶

From here, we need to

Clean and check for possible missing values
Merge year, month, day and ad_time into one column named timestamp

Possible optimizations are

Using label encoding in categorical columns
Use better datatypes for specific purposes, like georeferenced values

Clean missing values¶

In [8]:

# total na numbers
df.isna().sum()

Out[8]:

FID           0
YEAR          0
MONTH         0
DAY           0
AD_TIME       0
BTID          0
NAME          0
LAT           0
LONG          0
WIND_KTS      0
PRESSURE      0
CAT           0
BASIN         0
Shape_Leng    0
dtype: int64

In [9]:

df.isnull().sum()

Out[9]:

FID           0
YEAR          0
MONTH         0
DAY           0
AD_TIME       0
BTID          0
NAME          0
LAT           0
LONG          0
WIND_KTS      0
PRESSURE      0
CAT           0
BASIN         0
Shape_Leng    0
dtype: int64

Okay, not so much to do here 😌

Merge date columns into a single one¶

We'll be using pd.to_datetime from https://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html

We'll need to convert AD_TIME from the current format to a valid datetime

In [36]:

df['AD_TIME'].head()

Out[36]:

0    1800Z
1    1200Z
2    0600Z
3    0600Z
4    1200Z
Name: AD_TIME, dtype: object

We join everything together in one column, as a dumb string. This is to make the conversion easier afterwards:

In [37]:

df['TIMESTAMP'] = df['YEAR'].astype(str) + \
                  df['MONTH'].astype(str).str.zfill(2) + \
                  df['DAY'].astype(str).str.zfill(2) + \
                  df['AD_TIME']

In [38]:

df['TIMESTAMP'].head()

Out[38]:

0    195708081800Z
1    196110031200Z
2    196208290600Z
3    196707140600Z
4    197208161200Z
Name: TIMESTAMP, dtype: object

We convert the dumb string column to a valid timestamp object, and we pass a format and we tell pandas that the time is in UTC:

In [39]:

df['TIMESTAMP'] = pd.to_datetime(df['TIMESTAMP'], format='%Y%m%d%H%MZ', utc=True)

In [40]:

df['TIMESTAMP'].head()

Out[40]:

0   1957-08-08 18:00:00+00:00
1   1961-10-03 12:00:00+00:00
2   1962-08-29 06:00:00+00:00
3   1967-07-14 06:00:00+00:00
4   1972-08-16 12:00:00+00:00
Name: TIMESTAMP, dtype: datetime64[ns, UTC]

We drop the other columns associated with time:

In [41]:

df = df.drop(['YEAR','MONTH','DAY','AD_TIME'], axis=1)

In [42]:

df.head()

Out[42]:

	FID	BTID	NAME	LAT	LONG	WIND_KTS	CAT	BASIN	Shape_Leng	TIMESTAMP
0	2001	63	NOTNAMED	22.5	-140.0	50	TS	Eastern Pacific	1.140175	1957-08-08 18:00:00+00:00
1	2002	116	PAULINE	22.1	-140.2	45	TS	Eastern Pacific	1.166190	1961-10-03 12:00:00+00:00
2	2003	124	C	18.0	-140.0	45	TS	Eastern Pacific	2.102380	1962-08-29 06:00:00+00:00
3	2004	168	DENISE	16.6	-139.5	45	TS	Eastern Pacific	2.121320	1967-07-14 06:00:00+00:00
4	2005	251	DIANA	18.5	-139.8	70	H1	Eastern Pacific	1.702939	1972-08-16 12:00:00+00:00

Further optimizations¶

These are valid for those columns that are clearly categorical, CAT and BASIN. We could encode this columns using label encoding which would allow us to store integers instead of long strings.

We could create another table that stores the meanings of the codes, this is more or less called a one to one relationship and it is discussed in here

The pros and cons of this approach are

Pros

It's more efficient to store and query integer values
We could create auxiliar tables that map the labels to their meanings, thus allowing for more atomic queries
This would allow for good normalizations practices that will help in scaling our database (and keeping it sane to maintain)

Cons

Integers lose any semantic meaning without a conversion table
Adding semantic meaning from an auxiliar table requires a JOIN operation, which is costly
It requires a bit more of effort
It could happen that the column is not categorical at all (i.e. more different values for BASIN show up in the future)

We will not use this approach since it beats the purpose of the problem, and for this case is overkill since each category does not have or need a full model by itself, since it would consist of only one attribute. We'll keep it simple.

This Stackoverflow question deals with normalization on one-to-one relationships:

If it fits within the rules of normalization, then 1:1 relationships can be normalized (by definition!) - In other words, there is nothing about 1:1 relationships that make it impossible for them to obey the normal forms.

To answer your question about the practicality of 1:1 relationships, there are times when this is a perfectly useful construct, such as when you have subtypes with distinct predicates (columns).

The reasons you would use 1:1 relationships depend on your point of view. DBAs tend to think of everything as being a performance decision. Data modelers and programmers tend to think of these decisions as being design or model oriented. In fact, there is a lot of overlap between these points of view. It depends on what your perspectives and priorities are. Here are some examples of motivations for 1:1 relationships:

You have some subset of columns that are very wide and you want to segregate them physically in your storage for performance reasons.

You have some subset of columns that are not read or updated frequently and you want to keep them apart from the frequently used columns for performance reasons.

You have some columns that are optional in general but they are mandatory when you know that the record is of a certain type.

You have some columns that logically belong together for a subtype and you want to model them to fit well with your code's object model.

You have some columns that can only apply to some subtype(s) of an entity super-type, and you want your schema to enforce the absence of this data for other subtypes.

You have some columns that belong to an entity but you need to protect these particular columns using more restrictive access rules (e.g. salary on an employee table).

So you can see, sometimes the driver is performance, sometimes it is model purity, or just a desire to take full advantage of declarative schema rules.

We could even push this further and say that the BASIN column is more of a boolean value since it has two values max. This allows us to rewrite the column in something like IS_BASIN_EASTERN_PACIFIC. We could say then that for those values where this is False then it means the Basin it's North Atlantic.

But this is never a good approach since it uses this information implicitly (thus hidden, and we don't want hidden stuff!), plus this will break the second a new category shows up.

In [43]:

df['BASIN'].astype('category').cat.codes.unique()

Out[43]:

array([0, 1])

In [44]:

df['CAT'].astype('category').cat.codes.unique()

Out[44]:

array([10,  1,  9,  3,  2,  6,  4,  5,  0, 11,  8,  7])

In [46]:

df.columns = df.columns.str.upper()

In [47]:

df.head()

Out[47]:

	FID	BTID	NAME	LAT	LONG	WIND_KTS	CAT	BASIN	SHAPE_LENG	TIMESTAMP
0	2001	63	NOTNAMED	22.5	-140.0	50	TS	Eastern Pacific	1.140175	1957-08-08 18:00:00+00:00
1	2002	116	PAULINE	22.1	-140.2	45	TS	Eastern Pacific	1.166190	1961-10-03 12:00:00+00:00
2	2003	124	C	18.0	-140.0	45	TS	Eastern Pacific	2.102380	1962-08-29 06:00:00+00:00
3	2004	168	DENISE	16.6	-139.5	45	TS	Eastern Pacific	2.121320	1967-07-14 06:00:00+00:00
4	2005	251	DIANA	18.5	-139.8	70	H1	Eastern Pacific	1.702939	1972-08-16 12:00:00+00:00

We look back at our table, and we rearrange the columns in the dataframe to match this table

fid INT PRIMARY KEY,
timestamp TIMESTAMP WITH TIME ZONE,
btid SMALLINT,
name VARCHAR,
lat DECIMAL(8,6),
long DECIMAL(9,6),
wind_kts SMALLINT,
pressure SMALLINT,
cat VARCHAR(2),
basin VARCHAR(16),
shape_length DECIMAL(7, 6)

In [48]:

# we rearrange our columns and reset the dataframe's index
df_csv = df[['FID', 'TIMESTAMP', 'BTID', 'NAME', 'LAT', 'LONG', 'WIND_KTS', 'PRESSURE', 'CAT', 'BASIN', 'SHAPE_LENG']].copy()
df_csv = df_csv.set_index('FID', drop=True)
df_csv.head()

Out[48]:

	TIMESTAMP	BTID	NAME	LAT	LONG	WIND_KTS	PRESSURE	CAT	BASIN	SHAPE_LENG
FID
2001	1957-08-08 18:00:00+00:00	63	NOTNAMED	22.5	-140.0	50	0	TS	Eastern Pacific	1.140175
2002	1961-10-03 12:00:00+00:00	116	PAULINE	22.1	-140.2	45	0	TS	Eastern Pacific	1.166190
2003	1962-08-29 06:00:00+00:00	124	C	18.0	-140.0	45	0	TS	Eastern Pacific	2.102380
2004	1967-07-14 06:00:00+00:00	168	DENISE	16.6	-139.5	45	0	TS	Eastern Pacific	2.121320
2005	1972-08-16 12:00:00+00:00	251	DIANA	18.5	-139.8	70	0	H1	Eastern Pacific	1.702939

We'll save our ammended csv somewhere else to be used later

In [49]:

df_csv.to_csv(path_or_buf='data/cleaned_files.csv', sep=';')

Creating the table with SQL¶

Next step is actually creating the table

In [3]:

import psycopg2

conn = psycopg2.connect(
    dbname="ihw", user="data_prod_user", password="changeme", host="192.168.32.3"
)

conn.autocommit = True
cur = conn.cursor()

# year, month, day, ad_time will be stored as timestampz
print("DROP TABLE IF EXISTS hurricanes")
cur.execute("DROP TABLE IF EXISTS hurricanes")

CREATE_TABLE_SQL ="""
CREATE TABLE hurricanes (
    fid INT PRIMARY KEY,
    timestamp TIMESTAMP WITH TIME ZONE,
    btid SMALLINT,
    name VARCHAR,
    lat DECIMAL(8,6),
    long DECIMAL(9,6),
    wind_kts SMALLINT,
    pressure SMALLINT,
    cat VARCHAR(2),
    basin VARCHAR(16),
    shape_length DECIMAL(8, 6)
    )
"""

print(CREATE_TABLE_SQL)

cur.execute(
    CREATE_TABLE_SQL
)

conn.commit()
print("Done.")
conn.close()

DROP TABLE IF EXISTS hurricanes

CREATE TABLE hurricanes (
    fid INT PRIMARY KEY,
    timestamp TIMESTAMP WITH TIME ZONE,
    btid SMALLINT,
    name VARCHAR,
    lat DECIMAL(8,6),
    long DECIMAL(9,6),
    wind_kts SMALLINT,
    pressure SMALLINT,
    cat VARCHAR(2),
    basin VARCHAR(16),
    shape_length DECIMAL(8, 6)
    )

Done.

Insert the data in the database¶

Assuming we have the CSV available in disk, we can load it onto the table using psycopg's copy_expert()

In [4]:

import psycopg2

conn = psycopg2.connect(
    dbname="ihw", user="data_prod_user", password="changeme", host="192.168.32.3")

conn.autocommit = True
cur = conn.cursor()

with open('data/cleaned_files.csv', 'r+') as f:
    cur.copy_expert("COPY hurricanes FROM STDIN WITH CSV HEADER DELIMITER ';'", f)
    
conn.close()

Grant specific privileges between users and tables¶

With our users and our table ready, we need to restrict the privileges some users have. This is so we can avoid unfortunate events like undesired DELETE or DROP instructions. Accidents happen.

In [17]:

import psycopg2

conn = psycopg2.connect(
    dbname="ihw", user="postgres", password="postgres", host="192.168.32.3"
)

conn.autocommit = True
cur = conn.cursor()

# revoke all privileges and add back specific ones
# on user data_prod_user
cur.execute("REVOKE ALL ON hurricanes FROM data_prod_user")
cur.execute("GRANT SELECT, INSERT, UPDATE ON hurricanes TO data_prod_user")

# on group readonly
cur.execute("REVOKE ALL ON hurricanes FROM readonly")
cur.execute("GRANT SELECT ON hurricanes TO readonly")

conn.commit()
print("Done.")
conn.close()

Done.

Verify permissions and roles¶

As `postgres` user¶

In [6]:

import psycopg2

conn = psycopg2.connect(
    dbname="ihw", user="postgres", password="postgres", host="192.168.32.3"
)

conn.autocommit = True
cur = conn.cursor()
cur.execute('SELECT * FROM hurricanes LIMIT 1')

print(cur.fetchall())

conn.close()

[(2001, datetime.datetime(1957, 8, 8, 18, 0, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None)), 63, 'NOTNAMED', Decimal('22.500000'), Decimal('-140.000000'), 50, 0, 'TS', 'Eastern Pacific', Decimal('1.140175'))]

As `data_prod_user`¶

In [19]:

import psycopg2

conn = psycopg2.connect(
    dbname="ihw", user="data_prod_user", password="changeme", host="192.168.32.3"
)

conn.autocommit = True
cur = conn.cursor()

cur.execute('SELECT * FROM hurricanes LIMIT 1')
cur.fetchall()

Out[19]:

[(2001,
  datetime.datetime(1957, 8, 8, 18, 0, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None)),
  63,
  'NOTNAMED',
  Decimal('22.500000'),
  Decimal('-140.000000'),
  50,
  0,
  'TS',
  'Eastern Pacific',
  Decimal('1.140175'))]

In [22]:

# we should be able to insert data
cur.execute('INSERT INTO hurricanes (fid) values (1000001)')
conn.commit()

In [25]:

# we can read this new row from the table
cur.execute('SELECT * FROM hurricanes WHERE fid = 1000001')
cur.fetchall()

Out[25]:

[(1000001, None, None, None, None, None, None, None, None, None, None)]

In [26]:

# but we should not be able to delete data
cur.execute('DELETE FROM hurricanes WHERE fid=1000000')
conn.commit()

---------------------------------------------------------------------------
ProgrammingError                          Traceback (most recent call last)
<ipython-input-26-06d32ae504fc> in <module>
      1 # but we should not be able to delete data
----> 2 cur.execute('DELETE FROM hurricanes WHERE fid=1000000')
      3 conn.commit()

ProgrammingError: permission denied for relation hurricanes

In [28]:

cur.execute("UPDATE hurricanes SET name = 'fakename' WHERE fid=1000001")
conn.commit()

In [30]:

# we read the updated changes back
cur.execute('SELECT * FROM hurricanes WHERE fid = 1000001')
cur.fetchall()

Out[30]:

[(1000001, None, None, 'fakename', None, None, None, None, None, None, None)]

In [31]:

conn.close()

Checking permissions as `analyst1`¶

In [32]:

import psycopg2

conn = psycopg2.connect(
    dbname="ihw", user="analyst1", password="changeme", host="192.168.32.3"
)

conn.autocommit = True
cur = conn.cursor()

cur.execute('SELECT * FROM hurricanes LIMIT 1')
cur.fetchall()

Out[32]:

[(2001,
  datetime.datetime(1957, 8, 8, 18, 0, tzinfo=psycopg2.tz.FixedOffsetTimezone(offset=0, name=None)),
  63,
  'NOTNAMED',
  Decimal('22.500000'),
  Decimal('-140.000000'),
  50,
  0,
  'TS',
  'Eastern Pacific',
  Decimal('1.140175'))]

In [33]:

# we should be able to insert data
cur.execute('INSERT INTO hurricanes (fid) values (1000001)')
conn.commit()

---------------------------------------------------------------------------
ProgrammingError                          Traceback (most recent call last)
<ipython-input-33-afc5d0bde908> in <module>
      1 # we should be able to insert data
----> 2 cur.execute('INSERT INTO hurricanes (fid) values (1000001)')
      3 conn.commit()

ProgrammingError: permission denied for relation hurricanes

In [34]:

cur.execute("UPDATE hurricanes SET name = 'fakename' WHERE fid=1000001")
conn.commit()

---------------------------------------------------------------------------
ProgrammingError                          Traceback (most recent call last)
<ipython-input-34-b52c899b5a8c> in <module>
----> 1 cur.execute("UPDATE hurricanes SET name = 'fakename' WHERE fid=1000001")
      2 conn.commit()

ProgrammingError: permission denied for relation hurricanes

In [35]:

# but we should not be able to delete data
cur.execute('DELETE FROM hurricanes WHERE fid=1000000')
conn.commit()

---------------------------------------------------------------------------
ProgrammingError                          Traceback (most recent call last)
<ipython-input-35-06d32ae504fc> in <module>
      1 # but we should not be able to delete data
----> 2 cur.execute('DELETE FROM hurricanes WHERE fid=1000000')
      3 conn.commit()

ProgrammingError: permission denied for relation hurricanes

Final words¶

There's not so much we can say on top of what has been shown with code. We can achieve almost anything with the psycopg2 driver, which is the most used python driver for postgres databases (their words, not mine).

Moving forward¶

We can add another abstraction layer on top of the driver, namely an ORM or object relational mapper. I use SQLAlchemy in my daily tasks and it has proven useful in many ways, but that's something for another topic.

Flask contexts (And how to use them)

2018-11-24T13:14:00-06:00

Motivation

There's this thing going on with the flask context and the request context, that used to confuse me a lot. Before testing these two probably will never be used, but after reading the documentation and writing your first tests, it's probable that you run into something like

Flask.url_for() error: Attempted to generate a URL without the application context being pushed

So, what is that went wrong?

Someone already wrote an exhaustive revision of what are the different approaches, but it actually never closes the issue of how to test an application properly.

Others mention it, but it just doesn't stick

For the sake of completeness and for my own understanding of the issue, here's my view on the topic.

Flask Contexts

In the lifecycle of a request, both an Application context and a Request Context are created at the beginning, and destroyed at the end (see Advanced Flask Patterns - Speaker Deck, Slide 7)
The idea behind having two detached contexts is so an application can exist outside of a request, and it's more of a design pattern (That a lot of people seems to hate) and it was different in previous versions of Flask as discussed in this SO question (Which is also a great discussion about the internals of Flask).

In short, context locals can be summarized as (as shown in Flask for Fun and Profit, Slide 27)

A pushed app context points the current app in use to current_app, and it gives meaning to other proxies that only make sense for some parameters in a live app instance, like url_for and g
A request context is more expensive and maps the request proxy to the current request in process
Every request pushes a new application context

Flask contexts vernacular

For a given app object, there are some design aspects and specific objects it's good to get familiar with when testing with Flask

`flask.app.test_client()`

Provides a client that can perform requests to our application.

`flask.app.app_context()`

The application context, it gives life to current_app. Starts and dies with a request.

`flask.app.test_request_context()`

The request context, it gives life to request. If there's no application context at the moment, it pushes a new one. Starts and ends with a request.

`g`

A proxy that lives within a pushed application context, used to store non sensible information about the current application. Its life is bound to that of the request.

`session`

A proxy that lives within a pushed request context, used to store sensible information and encrypted with your SECRET_KEY.

Practical differences (in code)

This SO answer puts it simple, so I will be kinda adding my own comments to it.

from flask import Flask, g
app = Flask(__name__)

with app.app_context():
    print('in app context, before first request context')
    print('setting g.foo to abc')
    g.foo = 'abc'
    print('g.foo should be abc, is: {0}'.format(g.foo))

    with app.test_request_context():
        # this reuses g from the current context
        print('in first request context')
        print('g.foo should be abc, is: {0}'.format(g.foo))
        print('setting g.foo to xyz')
        # this is the same g, so it will be replaced
        g.foo = 'xyz'
        print('g.foo should be xyz, is: {0}'.format(g.foo))

    print('in app context, after first request context')
    print('g.foo should be abc, is: {0}'.format(g.foo))

    with app.test_request_context():
        print('in second request context')
        print('g.foo should be abc, is: {0}'.format(g.foo))
        print('setting g.foo to pqr')
        g.foo = 'pqr'
        print('g.foo should be pqr, is: {0}'.format(g.foo))

    print('in app context, after second request context')
    print('g.foo should be abc, is: {0}'.format(g.foo))

And here's the output that it gives:

in app context, before first request context
setting g.foo to abc
g.foo should be abc, is: abc
in first request context
g.foo should be abc, is: abc
setting g.foo to xyz
g.foo should be xyz, is: xyz
in app context, after first request context
g.foo should be abc, is: xyz
in second request context
g.foo should be abc, is: xyz
setting g.foo to pqr
g.foo should be pqr, is: pqr
in app context, after second request context
g.foo should be abc, is: pqr

In this first example g is shared across contexts because it's the same context in nested contexts, and it's a caveat mentioned in the answer (some emphasis is mine)

"Every request pushes a new application context". And as the Flask docs say, the application context "will not be shared between requests". Now, what hasn't been explicitly stated (although I guess it's implied from these statements), and what my testing clearly shows, is that you should never explicitly create multiple request contexts nested inside one application context, because flask.g (and co) doesn't have any magic whereby it functions in the two different "levels" of context, with different states existing independently at the application and request levels.

The reality is that "application context" is potentially quite a misleading name, because app.app_context() is a per-request context, exactly the same as the "request context". Think of it as a "request context lite", only required in the case where you need some of the variables that normally require a request context, but you don't need access to any request object (e.g. when running batch DB operations in a shell script). If you try and extend the application context to encompass more than one request context, you're asking for trouble. So, rather than my test above, you should instead write code like this with Flask's contexts:

from flask import Flask, g
app = Flask(__name__)

with app.app_context():
    print('in app context, before first request context')
    print('setting g.foo to abc')
    g.foo = 'abc'
    print('g.foo should be abc, is: {0}'.format(g.foo))

with app.test_request_context():
    print('in first request context')
    print('g.foo should be None, is: {0}'.format(g.get('foo')))
    print('setting g.foo to xyz')
    g.foo = 'xyz'
    print('g.foo should be xyz, is: {0}'.format(g.foo))

with app.test_request_context():
    print('in second request context')
    print('g.foo should be None, is: {0}'.format(g.get('foo')))
    print('setting g.foo to pqr')
    g.foo = 'pqr'
    print('g.foo should be pqr, is: {0}'.format(g.foo))

Which will give the expected results:

in app context, before first request context
setting g.foo to abc
g.foo should be abc, is: abc
in first request context
g.foo should be None, is: None
setting g.foo to xyz
g.foo should be xyz, is: xyz
in second request context
g.foo should be None, is: None
setting g.foo to pqr
g.foo should be pqr, is: pqr

In this second example, all three with blocks are pushing independent contexts. The difference is that last two are also pushing an application context implicitly, and the first one is not pushing a request context.

So, how do I test my app?

I will be using pytest because it's good. Just google "unittest versus pytest" to find out why.

This SO answer summarizes the differences in the best way I've found so far.

If you want to make a request to your application, use the test_client.
c = app.test_client()
response = c.get('/test/url')
# test response
If you want to test code which uses an application context (current_app, g, url_for), push an app_context.
with app.app_context():
    # test your app context code
If you want test code which uses a request context (request, session), push a test_request_context.
with current_app.test_request_context():
    # test your request context code

It's also important to note that

Both app and request contexts can also be pushed manually, which is useful when using the interpreter.
>>> ctx = app.app_context()
>>> ctx.push()

So far there are two approaches:

push an application context at the beginning inside a fixture, and pop it at the end
pass an application object from the fixture and produce contexts locally inside each test

Everything in the middle is prone to produce weird results and drive you crazy, so try to be consistent!

Armin Ronacher suggests a pytest fixture pattern in this presentation (slides), which relies in the first approach. The code is as follows:

import pytest

@pytest.fixture(scope="module")
def app(request):

    # import app factory pattern
    from yourapp import create_app
    app = create_app()

    # pushes an application context manually
    ctx = app.app_context()
    ctx.push()

    # bind the test life with the context through the
    request.addfinalizer(ctx.pop)
    return app

@pytest.fixture()
def test_client(request, app):

    client = app.test_client()
    client.__enter__()

    request.addfinalizer(
        lambda: client.__exit__(None, None, None))
    return client

Note that request here refers to the pytest fixture

But how do I test? really

After a lot of churning and with the help of #pocoo in IRC, I came up with this:

import pytest


@pytest.fixture
def runner(app):
    """A test runner for the app's Click commands."""
    return app.test_cli_runner()


@pytest.fixture(scope='session')
def app(request):
    from yourapp import create_app
    return create_app('testing')


@pytest.fixture(autouse=True)
def app_context(app):
    """Creates a flask app context"""
    with app.app_context():
        yield app


@pytest.fixture
def request_context(app_context):
    """Creates a flask request context"""
    with app_context.test_request_context():
        yield


@pytest.fixture
def client(app_context):
    return app_context.test_client(use_cookies=True)


@pytest.fixture(scope="model")
def db(app_context):

    # extensions pattern explained in here https://stackoverflow.com/a/42910185/5819113
    from yourapp.extensions import db

    db.create_all()

    # seed the database
    seed_db()

    yield db

    # teardown database
    # https://stackoverflow.com/a/18365654/5819113
    db.session.remove()
    db.drop_all()
    db.get_engine(app_context).dispose()


def seed_db():
    # insert default users and roles
    print("Seeding the database or something")

This is working for me. In short

If your test requires a living app, import the app_context fixture
If your test requires a populated database, import db
If your test requires examining a part of the request, import the request_context fixture
If your test requires a logged in user, you can use the client fixture and flask_login as

def login(client, email, password):
    """A helper method that logs in users"""
    return client.post(url_for('auth.login'),
                    data={
                        'email': email,
                        'password': password},
                    follow_redirects=True)


def test_with_logged_in_user(client):
    """Tests a view that requires authentication"""
    from flask_login import current_user

    # client is a pytest fixture
    with client as c:
        # login is a helper method that issues a POST request
        login(c, "user@test.com", "test")
        assert current_user.email == "user@test.com"
        assert current_user.is_authenticated

        response = c.get(
            '/restricted_view/',
            follow_redirects=True)

        assert response.status_code == 200
        assert b'Stuff that should appear in your view' in response.data

That's all I have to say about it.

On getting started with Jekyll

2017-10-20T18:01:00-06:00

I've decided to start a blog, (again, for the nth time) and this time I am giving a try on jekyll and the github pages, since they seem to do quite a fine job.

Setting things up was easy. I'm not going to copy and paste the guides already present in both sites, since it won't add anything. Instead I will try to share a bit of the troubleshooting I had to do during the process.

First steps

Okay, it took me about nothing to have my site showing a humble Hello World in my index.html on my site.

Creating your personal page

You can create a site for yourself or for some repo you own. I'm dealing with the first case for now: A site for myself.

Long story short, what I did was:

Have a Github account. Duh. Then create a repo named [your username].github.io. In my case, my username is diegoquintanav. You can change that later, it seems. Let's not care about that for now.
Add a file named index.html in the root folder of the repo, and write Hello World in it.
???
Profit! go to [your username].github.io and you should be able to see the words too.

That was a piece of cake. Next thing you can do is to set up Jekyll. This requires a bit more of work, but don't be scared. Before doing anything, I believe it's a good thing to understand why should you use Jekyll and not just plain HTML or Wordpress for this matter. Since this is a personal issue and a matter of preference, I can only share my experience and talk about my needs.

I tried using Wordpress, which has a lot of resources to get up and running right out of the box. It is also free, which is nice. In my case, I wanted to have more control over the stuff I was writing, namely be able to deploy my stuff whenever I want and to be able to change stuff from the very core, so to speak. Plus, this should work as a first experience running a site. Github pages may be too vanilla for the hardcore developer, but is a very balanced solution for enthusiasts like me.

Now there are two ways it seems, to get started. The first one implies creating the folder structure manually, and this process is more or less explained in this fine post which is also referenced by the Jekyll site.

What I didn't like about the post, though, is that doing things manually usually imply forgetting about things. I prefer to stick with the solution using Bundle and Jekyll from the terminal, which is more or less described here and here.

Issues

New posts are not being displayed

If you deployed Jekyll correctly, it should use the minima template by default. You should be able to add more posts inside the folder _posts and see them instantly on your localhost. That didn't happen in my case. A short way to debug this issue that solved my problem required the --verbose flag when firing up the service.

$ sudo bundle exec jekyll serve --verbose
  Logging at level: debug
Configuration file: /home/diego/Documents/diegoquintanav.github.io/_config.yml
  Logging at level: debug
         Requiring: jekyll-feed
         Requiring: kramdown
            Source: /path/to/diegoquintanav.github.io
       Destination: path/to/diegoquintanav.github.io/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
       EntryFilter: excluded /Gemfile
       EntryFilter: excluded /Gemfile.lock
           Reading: _posts/2017-10-27-on-jekyll.md
           Reading: _posts/2017-10-26-welcome-to-jekyll.markdown
          Skipping: _posts/2017-10-27-on-jekyll.md has a future date
        Generating: JekyllFeed::Generator finished in 0.000744857 seconds.
...

In line 15 you can observe the Skipping: _posts/2017-10-27-on-jekyll.md has a future date. Changing the date solved the problem. About the sudo thing, consider it a malpractice. The ruby environment manager rbenv fought bravely against the Anaconda's gcc dependencies and I just could not manage to have a ruby instance outside of root. Sorry.

Other reasons may involve a bad filename formatting or the presence of colons : in it. You can read more about this here.

Using Latex equations

This was not trivial for me. If you follow the guidelines inside the documentation, you should end in the tutorial.

It should be something easy. Copying and pasting the CDN in the <head> of the document. But where is this file?

This was not obvious to me.

After some google-fu, nothing was explicit enough to me. This post helped a bit, but after realizing that the docs were not good enough, I ended up in #jekyll on freenode.

With a bit of help of some nice anons, I found out that

Themes are either Gem-installed or locally installed. By default after deploying Jekyll, you should be using Minima as a Gem Theme. This means basically that the files that compose your theme live somewhere else. Run bundle show minima, which displays the path of the theme. Somewhere inside var if you are using Ubuntu. If you cd into that folder you should see some folders, namely _layouts, which are the templates of the theme and _includes which I'm not sure yet. Every theme has more or less the same folders with the exact names, but the way pages are composed by means of templates should differ. Expect that.
Inside the _layouts folder you should find head.html. Copy that file inside your own project, in a folder called _layouts if you don't have it yet. By doing this you are basically overriding the base files inside the Gem theme with your own.
Now you can edit it. Copy the script node from the extras site on Jekyll and there you go. You should be able to render$ \LaTeX$ equations easily.
- Please beware that in comparison to the traditional way, inline equations and almost every equation is rendered using doble dollar signs i.e.

$$\text{A } \LaTeX \text{ equation.}$$

$$\text{A } \LaTeX \text{ equation.}$$

Other implementation details depend on the Markdown compiler being used, in my case (and odds that yours too if you are starting with jekyll) I'm using kramdown which is set by default.

Next Steps

I think there are a lot of other things to play around, which I will document and comment in the future posts. Not yet. This is my MVP for now. Next iteration should include:

Comments section. Might prove useful somewhere.
Add more sections, editing the navbar or something equivalent. Add more endpoints.
Change the site's domain.
Add some google analytics sorcery or equivalent.
Theme editing and/or theme changing
Add embedded python REPL instances. I saw them somewhere on the internet. I want them.

Moving rocks around

A short DLT tutorial

The Pinochet-Rettig dataset, Revisited

What

Python and Docker

Update

What

Docker image requirements

Example project

Global variables

The builder stage

The app-pre stage

The app stage

The dev stage

Buiding using docker compose

Try out your new containers

Full Dockerfile

References

About docker and poetry

For more on users and groups

About renewing arguments at multiple stages

Pyenv, Poetry and Pipx: three python tools to rule them all

Update

Sharpen your axe

Pin those dependencies

Get control of your python environments

But first, what is a dependency?

What is a virtual environment?

Why do we need a virtual environment?

The easy win: using virtualenv

The "a bit more involved" win using pyenv + pipx + poetry

A demonstration using Docker

Writing your project as a CLI with poetry

Building and publishing your project

Publish to pypi

Publish to a private repository

Creating your own private repository using docker compose

Adding dependencies from private repositories to pyproject.toml

Managing virtual environments inside docker

Using dbt, fal and novu for forecasting pipelines

Pelican (and personal) update

Hug a rock today

Modeling the Pinochet Dataset using Neo4j

Introduction

Roadmap

Before starting

Tabular form data exploration

Related tutorials

On graphql

On neo4j

Modeling

First approach: Don't change anything

Concepts (or Nodes)

Perpetrators

Victims

ViolentEvent

More on modeling series of events

Relationships

Polishing things

Loading the dataset into a graph

General statistics

Using the geospatial module of neo4j

Which type of events are linked to more locations?

Longest paths

Number of people linked to a location

Londres 38

Wrapping up

Plotting traffic maps in Barcelona

Introduction¶

Hands on¶

Manipulating WKT data with shapely¶

Reshaping the strings¶

First plot using geopandas¶

Tweaking the plot¶

Plotting routes on top of Barcelona using Folium¶

Plotting traffic data¶

Quick survey on the data¶

How busy are trams compared to the expected traffic, on average, for the month of May?¶

How busy were trams on average, on an hour basis, per tram line, during the month of May?¶

Baking everything together¶

The `builder` stage

The `app-pre` stage

The `app` stage

The `dev` stage

The easy win: using `virtualenv`

The "a bit more involved" win using `pyenv` + `pipx` + `poetry`

Writing your project as a CLI with `poetry`

First plot using `geopandas`¶

Plotting routes on top of Barcelona using `Folium`¶

Managing database backups with `pg_dump`

As `postgres` user¶

As `data_prod_user`¶

Checking permissions as `analyst1`¶

`flask.app.test_client()`

`flask.app.app_context()`

`flask.app.test_request_context()`

`g`

`session`