When dealing with databases you will need the ability to store certain dates and/or timestamps in your tables.
Let’s find out how you can do that in an SQLite database.

SQL table creation

SQLite has the data type TIMESTAMP for storing date-times

CREATE TABLE social_media (
        social_media_id INTEGER,
        insertion_date TIMESTAMP,
        yt_subs INTEGER
        fb_pg INTEGER
        insta_pg INTEGER,
        twitter_pg INTEGER)

Python code

In our Python code we will have to add this parameter to the connection

detect_types=sqlite3.PARSE_DECLTYPES | sqlite3.PARSE_COLNAMES

db_connection = sqlite3.connect('bmv.db', detect_types=sqlite3.PARSE_DECLTYPES | sqlite3.PARSE_COLNAMES)
cursor = db_connection.cursor()
cursor.execute('''SELECT * FROM social_media ORDER BY insertion_date DESC''')
rows = cursor.fetchall()

Now we will get datetime objects from our query.

Further Reading

This article is part of our SQL tutorial.

The post How to deal with date and time in SQLite appeared first on Creatronix.

Meanwhile there are a bunch of articles, so time for a overview page:

1. 1. SQL-Basics: Create, Read, Update & Delete
   2. SQL-Basics: Relations
   3. SQL-Functions – SQL-Basics 3
   4. SQLite3: Python and SQL
   5. Subqueries: Update column with values from another column
   6. SQL – NULL-safe equals operator
   7. SQL – the dark side
   8. SQL comments
   9. How to deal with date and time in SQLite

The post SQL-Tutorial appeared first on Creatronix.

You need to use the binary numbers in the diagnostic report to generate two new binary numbers (called the gamma rate and
the epsilon rate). The power consumption can then be found by multiplying the gamma rate by the epsilon rate.

Each bit in the gamma rate can be determined by finding the most common bit in the corresponding position of all numbers
in the diagnostic report. For example, given the following diagnostic report:

00100
11110
10110
10111
10101
01111
00111
11100
10000
11001
00010
01010


Considering only the first bit of each number, there are five 0 bits and seven 1 bits. Since the most common bit is 1, the first bit of the gamma rate is 1.

The most common second bit of the numbers in the diagnostic report is 0, so the second bit of the gamma rate is 0.

The most common value of the third, fourth, and fifth bits are 1, 1, and 0, respectively, and so the final three bits of the gamma rate are 110.

So, the gamma rate is the binary number 10110, or 22 in decimal.

The epsilon rate is calculated in a similar way; rather than use the most common bit, the least common bit from each position is used.
So, the epsilon rate is 01001, or 9 in decimal. Multiplying the gamma rate (22) by the epsilon rate (9) produces the power consumption, 198.

Use the binary numbers in your diagnostic report to calculate the gamma rate and epsilon rate, then multiply them together. What is the power consumption of the submarine? (Be sure to represent your answer in decimal, not binary.)

Reading the data

import pandas as pd

df = pd.read_csv("./aoc_day_03_test_data.txt", dtype = str, header=None)
df.columns = ["original"]

We need to define the input data as string via dtype = str. Otherwise it will be treated as integer and we loose the leading zeroes.

Now we can split the strings into individual bits with the split() function.

To generate a new column for every bit we use the tolist() function. The first and the last column is empty, so let’s remove them.

df = pd.DataFrame(df["original"].str.split('').tolist())
del df[0]
df = df.iloc[:,:-1]
df

Now we have to look which value appears the most in each column.

df = pd.concat([df[column].value_counts() for column in df], axis = 1) 
df

The most common bits are

most_common_bits = df.idxmax()
most_common_bits

1    1
2    0
3    1
4    1
5    0
dtype: object

Now we have to concatenate these bits to one string and  convert it back into an integer

gamma_rate = ''.join(most_common_bits)
gamma_rate = int(gamma_rate, 2)
gamma_rate

That gives us the gamma rate of 22

The epsilon rate can be calculated by inverting all significant bits in the most_common_bits value. We do this by creating a bit mask

bit_mask = 2 ** len(most_common_bits) - 1

and applying it with an XOR to the gamma rate:

epsilon_rate = gamma_rate ^ bit_mask
epsilon_rate

The solution is:

gamma_rate * epsilon_rate

The post 2021 – Advent of code – Day 3 appeared first on Creatronix.

Today the puzzle got a bit trickier than Day 1.

The submarine seems to already have a planned course (your puzzle input). You should probably figure out where it's going. For example:

    forward 5
    down 5
    forward 8
    up 3
    down 8
    forward 2

Your horizontal position and depth both start at 0. The steps above would then modify them as follows:

    forward 5 adds 5 to your horizontal position, a total of 5.
    down 5 adds 5 to your depth, resulting in a value of 5.
    forward 8 adds 8 to your horizontal position, a total of 13.
    up 3 decreases your depth by 3, resulting in a value of 2.
    down 8 adds 8 to your depth, resulting in a value of 10.
    forward 2 adds 2 to your horizontal position, a total of 15.

After following these instructions, you would have a horizontal position of 15 and a depth of 10. (Multiplying these together produces 150.)

Calculate the horizontal position and depth you would have after following the planned course. What do you get if you multiply your final horizontal position by your final depth?

So Pandas here we go again:

df = pd.read_csv("./aoc_day_02_data.txt", delimiter=" ",header=None)
df.columns = ["command", "value"]

Alright, reading in the data and naming the columns are the same steps as yesterday. Now we have to columns.

horizontal = df[df['command']=="forward"]["value"].sum()

The horizontal value can be calculated with the sum function when we filter the data frame to rows where the command is “forward”

depth = df[df['command']=="down"]["value"].sum() - df[df['command']=="up"]["value"].sum()

The depth can calculated by summing up the down and up commands separately and subtract the sums from each other.

Now we have to multiply the depth and the position to get the solution

position = depth * horizontal

Part 2

    down X increases your aim by X units.
    up X decreases your aim by X units.
    forward X does two things:
        It increases your horizontal position by X units.
        It increases your depth by your aim multiplied by X.

Now, the above example does something different:

    forward 5 adds 5 to your horizontal position, a total of 5. Because your aim is 0, your depth does not change.
    down 5 adds 5 to your aim, resulting in a value of 5.
    forward 8 adds 8 to your horizontal position, a total of 13. Because your aim is 5, your depth increases by 8*5=40.
    up 3 decreases your aim by 3, resulting in a value of 2.
    down 8 adds 8 to your aim, resulting in a value of 10.
    forward 2 adds 2 to your horizontal position, a total of 15. Because your aim is 10, your depth increases by 2*10=20 to a total of 60.

After following these new instructions, you would have a horizontal position of 15 and a depth of 60. 
(Multiplying these produces 900.)

Using this new interpretation of the commands, calculate the horizontal position and depth you would have after following the planned course. 
What do you get if you multiply your final horizontal position by your final depth?

To get an overview I simplified the table

     a   d
f 5  0   0
d 5  5
f 8  5  40
u 3  2
d 8 10
f 2 10  20

Here I had a hard time to do it with pandas so vanilla python to the rescue:

if __name__ == '__main__':

    with open("./aoc_day_02_data.txt") as file:
        lines = file.readlines()
        lines = [line.rstrip() for line in lines]

    horizontal = 0
    current_aim = 0
    depth = 0
    for line in lines:
        print(line)
        command, value = line.split(" ")
        value = int(value)
        if command == "forward":
            horizontal += value
            depth += value * current_aim
        if command == "down":
            current_aim += value
        if command == "up":
            current_aim += value * -1

    print(f"horizontal: {horizontal}")
    print(f"depth: {depth}")
    print(horizontal * depth)

Update

I’ve figured out how to do it with Pandas as well

import pandas as pd

df = pd.read_csv("./aoc_day_02_test_data.txt", delimiter=" ",header=None)
df.columns = ["command", "value"]

horizontal = df[df['command']=="forward"]["value"].sum()

df.loc[df['command']=="up", "value"] = df[df['command']=="up"].mul(-1)
df["aim"] = 0

df.loc[df['command']!="forward", "aim"] = df[df['command']!="forward"]["value"]
df["current_aim"] = df["aim"].cumsum()

df.loc[df['command']=="forward", "depth"] = df[df['command']=="forward"]["value"] * df[df['command']=="forward"]["current_aim"]
depth = df["depth"].sum()

The post 2021 – Advent of code – Day 2 appeared first on Creatronix.

What is advent of code?

It’s an advent Calendar for programmers. You get 25 challenges starting December 1st. Caveat: you have to solve the challenge to be eligible for the next day’s challenge 🙂

Day 1 Challenge – Part 1

On the first day your first task is to count how many times a value is bigger than its predecessor. They give us some sample data

199 N/A
200 bigger
208 bigger
210 bigger
200 smaller
207 bigger
240 bigger
269 bigger
260 smaller
263 bigger

When we count the times a value is bigger we get seven times bigger.

The actual data contains 2000 rows. This isn’t exactly big data but I’ve wanted to dust off my Pandas skill, so here we go:

Let’s look at the data

import pandas as pd

df = pd.read_csv("./aoc_day_01_data.txt", header=None)
df.describe

With the read_csv() function we can read in our data file and convert it into a data frame. It’s important to hand over the header=None. Otherwise pandas assumes the first row is a column header.

df.describe gives us:

<bound method NDFrame.describe of          0
0      159
1      158
2      174
3      196
4      197
...    ...
1995  8538
1996  8543
1997  8545
1998  8557
1999  8568

[2000 rows x 1 columns]>

Because we want to reference the columns by name we add a column header

df.columns = ["original"]

To compare the nth cell with its n+1th cell neighbour be add a new column but shift the values

df['shifted'] = df['original'].shift(-1)

The output looks like this:

We add another column where we place the value True when the value from the current row in the shifted column is bigger than in the original column:

df['increased'] = (df['shifted'] > df['original'])

Now it starts to look like the sample data from the introduction:

the last thing we have to do is counting how many times True occurs:

true_count = df['increased'].sum()

which gives us “1583”

This is a bit of a hack because it assumes that True equals 1 and False == 0

A more elegant solution is to use value_counts:

df['increased'].value_counts(dropna=False)

No the output is:

True     1583
False     417
Name: increased, dtype: int64

And 1583 is the number we are looking for. This earned us our first golden star and unlocked the second part of the challenge:

Part 2

The second part is a bit more challenging because we have to sum up three adjacent values and compare them to the next three values.

199  A       
200  A B     
208  A B C   
210    B C D
200  E   C D
207  E F   D
240  E F G
269    F G H
260      G H
263        H

I created a new notebook and started like part 1 with reading the data and naming the first column

import pandas as pd

df = pd.read_csv("./aoc_day_01_data.txt", header=None)
df.columns = ["original"]

To add the sum of three values to the row of the first value we use the following code

indexer = pd.api.indexers.FixedForwardWindowIndexer(window_size=3)
df["rolling_sum"] = df.original.rolling(window=indexer).sum()

This demonstrates the power of Pandas once more: you have integrated sliding window functions!

The rest is equal to part one “shift, compare and count”

df['shifted_rs'] = df['rolling_sum'].shift(-1)
df['increased_rs'] = (df['shifted_rs'] > df['rolling_sum'])
true_count = df['increased_rs'].sum()
true_count

As a little Fingerübung I did the same with vanilla Python:

data = []
with open("./aoc_day_01_test_data.txt") as f:
    for line in f:
        data.append(int(line.rstrip()))

triplet_sums = []

for i, v in enumerate(data):
    if i < (len(data) - 2):
        triplet_sum = data[i] + data[i+1] + data[i+2]
        triplet_sums.append(triplet_sum)
print(triplet_sums)

sums_larger_than_previous_sums = 0
for i, v in enumerate(triplet_sums):
    if i < (len(triplet_sums) - 1):
        if triplet_sums[i] < triplet_sums[i+1]:
            sums_larger_than_previous_sums += 1

print(sums_larger_than_previous_sums)

Which works but is less elegant.

Stay tuned for more!

The post 2021 – Advent of code – Day 1 appeared first on Creatronix.

from sklearn.model_selection import KFold

kf = KFold(n_splits=2)

kf.split(df_train)

step = 0 # set counter to 0
for train_index, val_index in kf.split(df_train): # for each fold
    step = step + 1 # update counter

    print('Step ', step)

    features_fold_train = df_train.iloc[train_index, [4, 5]] # features matrix of training data (of this step)
    features_fold_val = df_train.iloc[val_index, [4, 5]] # features matrix of validation data (of this step) 

    target_fold_train = df_train.iloc[train_index, 6] # target vector of training data (of this step)
    target_fold_val = df_train.iloc[val_index, 6] # target vector of validation data (of this step) 

    print("VALIDATE:", val_index)
    print('Dimensions features matrix for validation: ', features_fold_val.shape)
    print("TRAIN:", train_index)
    print('Dimensions features matrix for training: ',features_fold_train.shape, '\n')

further info

The post k-fold crossvalidation with sklearn appeared first on Creatronix.

I’ve assembled some pandas code snippets

Reading Data

Reading CSV

import pandas as pd

# read from csv
df = pd.read_csv("path_to_file")

Can also be textfiles. file suffix is ignored

The default limiter for comma separated value files is the comma. If you have data with another delimiter you can specify it via:

delimiter=";"

If your data has no header you can pass header=None into the function

df = pd.read_csv("./aoc_day_01_data.txt", header=None)

With skiprows you can start reading in at any row

skiprows=8

Sometimes you need to alter the encoding as well:

encoding="cp1252"

Reading Excel

You can read excel files as well but you need to install

pip install openpyxl

df = pd.read_excel("./my_excel_sheet.xlsx")

With sheet_name you can select the individual sheet:

sheet_name="my_sheet_1"

Inspecting data

Basic information

df.describe()

Length

len(df)

showing entries

df.head()

df.tail(10)

Indexing

df['A']

gives you column A

iloc gives you entries based on numerical index

#      [row, column]
df.iloc[0,   0]

#     [row, column]
df.loc[:, :]

Data Cleaning

Dropping columns

del df["column_name"]

Renaming columns

df.columns = ["new_column_name", ...]

Comparing columns

df['increased'] = (df['shifted'] > df['original'])

Shifting columns

df['shifted'] = df['original'].shift(-1)

Splitting

Splitting strings into individual columns

df = pd.DataFrame(df["original"].str.split('').tolist())

Counting and Calculating

Summing columns

df["value"].sum()

Cumulative sum

df["aim"].cumsum()

Rolling sum

indexer = pd.api.indexers.FixedForwardWindowIndexer(window_size=3)
df["rolling_sum"] = df.original.rolling(window=indexer).sum()

Counting value occurence

df['increased'].value_counts(dropna=False)

Counting occurrences for all columns

df = pd.concat([df[column].value_counts() for column in df], axis = 1)

Convert column to datetime

df.loc[:, 'date'] = pd.to_datetime(df.loc[:, 'date'])

Convert datetime to minutes since midnight

df_train.loc[:, 'msm'] = df_train.loc[:, "date"].dt.hour * 60 + df_train.loc[:, "date"].dt.minute

The post Pandas Cheat Sheet appeared first on Creatronix.

Learning Data Science can be grueling and overwhelming sometimes.

When I feel too overwhelmed it’s time to draw a picture.

This my current overview of what a data scientist has to do:

General tools

Linear Algebra with numpy

numpy random choice

Numpy linspace function

Data acquisiton

Data Science Datasets: Iris flower data set

Data cleaning

Pandas Cheat Sheet

Data exploration

Introduction to Pandas

Feature Scaling

Feature Scaling

Estimator fit

Linear Regression with sklearn cheat sheet

Lesson 2: Naive Bayes

Lesson 3: Support Vector Machines

Validation

What is Cross-Validation in Data Science?

k-fold crossvalidation with sklearn

Visualization

Introduction to matplotlib

Introduction to matplotlib – Part 2

Introduction to matplotlib – Part 3

Scatterplot with matplotlib

Interpretation

Metrics about the quality of your model

Classification: Precision and Recall

Receiver Operating Characteristic

Confusion Matrix

The post Data Science Pipeline appeared first on Creatronix.

You can download the installer at https://julialang.org/

On Windows you have to set the path manually, e.g. C:\Julia-1.3.1\bin

Let’s fire up a command line and type: julia

You will be greeted with a nice little ascii art and the prompt.

If You want to close the prompt You can use CTRL + D

Hello Julia

To get over with, here is the hello world

julia> show("Hello Julia")
"Hello Julia"

Julia has a show function which gives you decorated text interpretations

But there is also a print function

julia> print("Hello Julia")
Hello Julia

Writing Programs

julia code is put into files with the file extension .jl

so let’s write the hello julia to a file

We can start it via

julia hello-julia.jl

Reading and Writing Files

open("data.txt") do file
    for line in eachline(file)
        println(line)
    end
end

asdf

open("data.txt") do file
    for line in enumerate(eachline(file))
        println(line[1], ": ", line[2])
    end
end

Reading CSV

As a developer with interest in data science I want to know what packages exist to handle data. The first one is the csv package to deal with comma separated values.

julia> import Pkg; Pkg.add("CSV")

julia>using CSV
julia>csv = CSV.read("./my_cars.csv")

5×4 DataFrames.DataFrame
│ Row │ Make   │  Model  │  Bought │  Sold │
│     │ String │ String  │ Int64   │ Int64 │
├─────┼────────┼─────────┼─────────┼───────┤
│ 1   │ Fiat   │  Punto  │ 2005    │ 2009  │
│ 2   │ Ford   │  Escort │ 2009    │ 2010  │
│ 3   │ Ford   │  Fusion │ 2003    │ 2012  │
│ 4   │ Audi   │  A5     │ 2012    │ 2099  │
│ 5   │ Honda  │  Jazz   │ 2015    │ 2099  │

Plots

import Pkg; Pkg.add("Plots")
using Plots
x = 1:10; 
y = rand(10); 
plot(x, y)

The post Introduction to the Julia programming language appeared first on Creatronix.

So let’s start with a real database like SQLite.

SQLite is a file based DBRMS and can be used for e.g. web sites. The official docs say:

“SQLite works great as the database engine for most low to medium traffic websites (which is to say, most websites). [..] Generally speaking, any site that gets fewer than 100K hits/day should work fine with SQLite.”

Because Knight Industries is not Google, Amazon nor Facebook we can definitely use SQLite.

Creating and connecting to a database

In Python it is pretty easy to connect to a SQLite database:

from sqlite3 import connect 
db_connection = connect('knight_industries.db')

If the file knight_industries.db does not exist, it will be created automagically. A nice little feature of the sqlite3 library.

But be careful: If You already have a database file and you mess up the path in the connect statement you will wonder why you cannot access your data, because a new file is created silently.

cursor = db_connection.cursor()
cursor.execute('''CREATE TABLE operatives (id INTEGER, name TEXT, birthday DATE)''')
cursor.execute('''INSERT INTO operatives (id, name, birthday) \
                  VALUES (1, "Michael Arthur Long", "1949-01-09")''')

db_connection.commit()
cursor.execute('''SELECT * FROM operatives''')
print cursor.fetchone()
db_connection.close()

The post SQLite3: Python and SQL appeared first on Creatronix.