Building Uptrixia

For the past while I've been working on a startup called Uptrixia. The problem it solves is one most people never see: across the world, online igaming domains get quietly blocked - by regulators, by internet providers, by deep-packet inspection at the network layer. An operator can be perfectly reachable from one country and completely invisible from the next, and often they only find out when their traffic drops and they have no idea why. There's no error message for "your provider in this country is filtering you." We're trying to turn that silence into something you can actually act on.

The way Uptrixia works is fairly direct. We probe domains continuously from many different networks and countries, watching for the moment something stops resolving the way it should. When a domain goes dark somewhere, we can tell the operator where it's blocked, which network or provider is doing the blocking, and give them a path to fix it. It sounds simple, but the interesting part is in the details - distinguishing a real block from ordinary downtime, covering enough vantage points to be trustworthy, and presenting it all so a non-engineer can understand it at a glance. A lot of the work lately has been in exactly those unglamorous places.

It's still early, and we're a small team, which I've come to appreciate more than I expected. Decisions are fast, the feedback loop is short, and everyone owns real pieces of the thing. I don't have grand conclusions to offer yet - mostly I wanted to write down where things stand and keep a record as we go. More to come as the product takes shape; for now it's heads-down, building.

If any of this resonates or you're curious, you can find the project at https://uptrixia.com. 

MySQL tips and tricks

Restore the DB

If you have sql.gz archive

1 - create target database on server

2 - zcat mydb.sql.gz | mysql -u root -h localhost -p mydb

Find the data location

1 connect to mysql;

2 select @@datadir

Micro Continuous Deployment System

Hi Everyone,

Hope you have a great day, Today we going to build the micro deployment system.

What we need:
- ubuntu server (root access)
- git repository with your code

1. SSH to your favorite Ubuntu server.

ssh -p 22 shmalex@ubutunserver

2. Clone project

git clone https://github.com/shmalex/wifispymap.git

3. Check if your project is working and configure everything you need.

3.1 The main trick in the successful deployment of the system - all configurations should be in a file that does not see LIVE.

This can be done in many ways:

1. place the configuration in the directory above
2. keep all the plates in the same environment
3. keep you configuration in DB.... (but you need to store the connection string to the db...)

3.2 Double check that you projects folder is 'clean' by running command:

git status

3.4 It should show you that there are No/0/ZERO changes

3.4 By knowing that not all projects are perfect, I might need to keep making changes in your project and repeat step 2 and 3 until you achieve the resuls in 3.4.

All good?

ok, now the Continues Deployment part

When you have done all steps in 1,2,3 - the CD is very easy, let's make scheduler that run the some script, that script will contain the commands to pull the project, and other commands that makes our project apply thous changes.

4 - Create 'Deployment' script:

# open home folder
$ cd
# craete file
$ echo '#!/bin/bash' > deployment.sh
# edit the file
$ nano deployment.bash

Ok, now the tricky part.

we need to specify full path to the folders and files in bash script. we can not use the '~/wifispymap' path.

this is my script: 

#!/bin/bash
cd /home/shmalex/wifispymap/
git pull

4.2 make that file executable

chmod +x deployment.sh 

Now let's test it!

1 Test permissions - 

run the command 

$ ls -ahl | grep 'deployment.sh'

It will show the file's permission, we are interested in the x

-rwxrwxr-x  1 shmalex shmalex   50 Jun 15 14:22 deployment.sh

#2 test from home directory

$ cd$ deployment.sh

Already up to date.

#3 test from the root

$ cd /
$ /home/shmalex/deployment.sh

Already up to date.

100% passed.

5 - Convert "Deployment" into ''Continuous Deployment"

We need to use the scheduler,

$ sudo crontab -e

Add new line that will run our script every 5 minutes

*/5 * * * * /home/shmalex/deployment.sh

5.1. Test, following line will show you your changes

$ sudo crontab -l 

5.1 Test

Scheduler adds lines to the syslog file, where we can find our deployment script

$ grep CRON /var/log/syslog

We almost done!!! :)

6. And last test - let's make changes and find them on our server :)

6.1 - make any changes in project, i will update the readme.md file

6.2 - monitor the cron job with command

$ grep CRON /var/log/syslog

...

Jun 15 14:40:01 shmalex-dellpc CRON[14218]: (root) CMD (/home/shmalex/deployment.sh)

Jun 15 14:40:03 shmalex-dellpc CRON[14217]: (CRON) info (No MTA installed, discarding output)

... 

6.3. check the file with text editor, or by running the ls -hls command to check the time of changes file.

Have fun! and feel free to ask any questions :)

Reading materials

1. https://ryanstutorials.net/bash-scripting-tutorial/bash-script.php

2. https://askubuntu.com/questions/650387/how-do-i-create-a-task-scheduler-in-ubuntu

3. https://stackoverflow.com/questions/8779951/how-do-i-run-a-shell-script-without-using-sh-or-bash-commands

How to make Jupyter Notebook full screen

Problem:

The Jupyter Notebook has 900 pixels in width and don't use all the "potential" of your screen.

Especialy when you try to make a presentation of the big screen and you have to Zoom It. (CTRL + MOUSE_SCROLL-UP)

That is exactly where you need the "Full Potential" of you screen.

You play with styles in the Browser's Developer Tool, but this effort is not reusable.

Solution:
Use the Stylebot Extention for Chrome

This extension will save your css and apply it to site when you visit it.

Steps to use
1. Install it
2. Navigate Jupyter Notebook page
3. Open the Stylebot

4. At the bottom of the panel click "Edit CSS"

5. Copy Paste this:
#notebook-container {
    padding: 0px;
}
.prompt {
    min-width: 10ex;
}
div.container {
    width: 95%;
}

so it should look like this:

If you have trouble on "pasting" - just retype this css ;-)


DONE! now it looks way batter - enjoy your presentation!

FYI - the Jupyter Notebook page handles most of the keyboard shortcuts so it's could be tricky to work with Stylebot on JN page.
I recommend to adjust your styles in the Developer Tools and the copy&paste the result into Stylebot.

GL feel free to ask any questions.

WinDBG + VS + DevExpress

.loadby sos clr
.load E:\Diagnostics\sosex.dll
>>  run !bhi
.sympath srv*c:\mss*http://msdl.microsoft.com/download/symbols
.sympath+ E:\WebsiteCode\Live\Platform\Velocity_DigitalHandler\NEWHANDLER\bin

.symfix
.reload /f /i
!sym noisy


!dlk (dead locks)
!dlk
!EEStack -- clr stack

Sampling Distribution

Sampling distribution

Imagine you have some data and you and you would like to know it's properties, like - mean, or average or something else - a Point Estimate. But you don't have a way to work with full data set (it's to large, or it's not fully available to you). But you have access to some portion of it.

To have an good aproximate value of that data you can make the Sampling Distribution.

The sampling distribution represents the distribution of the point estimates based on samples of a fixed size from a certain population.

 It is useful to think of a particular point estimate as being drawn from such a distribution.

To make that point more visible I made the set of tests to show that it really works - the sample size is our variable that experiment.

import mysql.connector
import pandas as pd
import numpy as np
from scipy.stats import bernoulli, binom, poisson, norm, uniform, beta
import matplotlib.pyplot as plt

Distributions and how to plot them with SciPy

Intro

To make the plotting is simple (not that painful), in that post I will make few plots with different distributions.

First importing the libraries. I will use the matplotlib for plotting the data. SciPy one of the core libraries for data science calculations:

import numpy as np # we will use it for generation the arrays
from scipy.stats import bernoulli, binom, poisson, norm, uniform, beta
import matplotlib.pyplot as plt

To make printing comfortable a made the printing function for Mean, Variance, Skew, Kurt:

def print_mvsk(*args):
    t = args[0]
    mean, var, skew, kurt, = float(t[0]),float(t[1]),float(t[2]),float(t[3])
    sd = np.sqrt(var)
    print(f'mean:{mean:.4f}\tvar:{var:.4f}\tskew:{skew:.4f}\nsd:{sd:.4f}\tkurt:{kurt:.4f}')

Performing Accurate Decimal Operations

In python the decimal calculations using the IEEE 754 algorithm.
>>> a = 4.1
>>> b = 2.2
>>> a + b
6.300000000000001

that means that
>>> (a + b) == 6.3 
False
Because python's float type stores data using the native representations.

If you want more accuracy Decimal class from decimal module can help you with that.

>>> from decimal import Decimal
>>> a = Decimal('4.1')
>>> b = Decimal('2.2')
>>> a + b
Decimal('6.3')
>>> (a + b) == Decimal('6.3')
True

Skype Admin

To get list of devices where you skype in used now just 
- open chat with anyone from your contact list
- type /showplaces and hit Enter

that command will show the list of "places" where you are signed it.

This what I get:

GL

10 useful linux commands

As part of the Galvanize Data Science Immersive program we have task to create the 10 linux command post. :)

Dask is a great tool to do stuff.'

but when you install it - can produce problems:

c:\ProgramData\Anaconda3\Lib\site-packages>py.test dask
============================= test session starts =============================
platform win32 -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: c:\ProgramData\Anaconda3\Lib\site-packages, inifile:
collected 3230 items / 1 errors / 5 skipped

=================================== ERRORS ====================================
__________ ERROR collecting dask/diagnostics/tests/test_profiler.py ___________
dask\diagnostics\tests\test_profiler.py:148: in
    pytest.param(lambda: ResourceProfiler(dt=0.01),
E   AttributeError: module 'pytest' has no attribute 'param'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!
===================== 5 skipped, 1 error in 24.40 seconds =====================

to fix that problem run:

pip install -U "pytest>=3.1.0"

ls - list files and directory in the folder
  ls -ahl
htop - shows the processes

mkdir - make a directiry

cd - change current directory

rmdir - remove directory

touch - make a file

find - search for file or directory

rm - remove file

stat - get file info

grep - find text in file
  grep --color -n -i "soap" *.csv
get the list of open ports
netstat -lntu

• l = only services which are listening on some port
• n = show port number, don't try to resolve the service name
• t = tcp ports
• u = udp ports
• p = name of the program

Security and Stuff

Default Password List

http://www.phenoelit.org/dpl/dpl.html

Short Cuts in Word

Batch tricks

1. How to compress files in folder

FOR %i IN (*.log) DO 7z.exe a "%~ni.7z" "%i"

Huge Memmory Working Set 18 Exabyte

Huge working set in 

Interesting what that would mean?

Process Explorer:

Task Manager shows:

Homemade Seabuckthorn Probiotic Drink

This post is about a kind of drink with which I share a long history: fermented drinks. They were a staple of my childhood diet and so I received their benefit all throughout my formative and adolescent years. Among the best fruit bases someone could use for this purpose is Sea Buckthorn, a fruit long renowned for its nutritional and medicinal virtues.< Read more on Body Bazaar Blog

Remove old drivers

Remove Old Drivers

To remove old and unused drivers from your Windows computer, first open Start Menu and right-click on Computer and select Properties. From the left side panel, click on Advanced system settings and click on Environment Variables. Alternatively, simply open Control Panel and type Environment Variables.

Now under the User variables box, click on New and type

devmgr_show_nonpresent_devices

in the Variable Name text box & 1 in the Variable Value  box. This will set a flag to allow you to now view unused devices.

You can also use this method to show non-present drivers.

Now type devmgmt.msc in start search and hit Enter to open the Device Manager.

Original: http://www.thewindowsclub.com/how-to-remove-old-unused-device-drivers-in-windows-7

CSS Learning Materials

Good report with new Features in CSS
"Always read drafts"
https://www.youtube.com/watch?v=VoA-aQu75Xk

На Всяки

PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4NCjxLZXlGaWxlPg0KCTxNZXRhPg0KCQk8VmVyc2lvbj4xLjAwPC9WZXJzaW9uPg0KCTwvTWV0YT4NCgk8S2V5Pg0KCQk8RGF0YT4vWUxYc05RVU5Qb25ZdmVEV3hYdUNmUkRsajM4bFM3M3R3UFkyakNkMElBPTwvRGF0YT4NCgk8L0tleT4NCjwvS2V5RmlsZT4NCg==

Как узнать ёмкость батареи на notebook'е (windows)

Работает для windows

1. запускаем cmd под правами администратора
2. набираем powercfg /energy
   ждем пока программа отработает
3. в этом же окне набираем energy-report.html

в открывшемся окне браузера ищем строки

Battery Information
вот что выдает программа о моем новом аккумуляторе (картинка)
2017 Jan 14

Старый Аккумулятор которому было уже за 4 года показывал значение Last Full Charge ~2500. Этого заряда хватало на 2 минутки.

Update 1

2018 Feb 22

потеря 13% ёмкости.

Список литературы:

• https://support.microsoft.com/en-us/kb/976034
• https://technet.microsoft.com/en-us/library/cc748940(v=ws.10).aspx
• https://blogs.msdn.microsoft.com/robmar/2009/05/18/powercfg-energy-windows-7-power-management-reports/
• http://www.hanselman.com/blog/PowerCfgTheHiddenEnergyAndBatteryToolForWindowsYoureNotUsing.aspx

Windows RPC Commands

Полезные toolzы для общения с сервером.

Общие Требования 

Нужен доступ к удленному серверу - windows account.

доменная ShmalexNET\Admin

или запись на самом сервере

192.168.24.24\Admin

.\192.168.24.24

Tasklist

tasklist /S 192.168.24.24 /u ShmalexNet\Admin /p 1234

Выдает список задача на сервере 192.168.24.24 и логиниться под пользователем Admin домена ShmalexNet с паролем 1234

Shutdown

shutdown /r /f /t 10 /m \\192.168.24.24

Перезагружаем /r удаленный сервер 192.168.24.24 принудительно /f с паузой в 10 секунд /t 10

iisreset

iisreset 192.168.24.24

Перезапускает серсив IIS на указанном сервере 192.168.24.24