Thursday, December 17, 2015

hubot + hangups + dokuwiki + zabbix

Fast post!

I have integrated dokuwiki and zabbix with hubot using hanhgups. This is a hell because you need to make a concrete calls with the hangups adapter.

I am using hungups api REST in hubot to receive the notifications. So we need to integrate that calls.



My shellscript template:



My Action:


Monday, November 16, 2015

Mydumper RPM v0.9.1 for CentOS 6.x (x86_64)

A week ago Mydumper jumped to the 0.9.1 with a lot of improvements. In the Percona blog we can read:

A significant change included in this version now enables Mydumper to handle all schema objects!!  So there is no longer a dependency on using mysqldump to ensure complex schemas are backed up alongside the data.
Let’s review some of the new features:
Full schema support for Mydumper/Myloader
Mydumper now takes care of backing up the schema, including Views and Merged tables. As a result, we now have these new associated options:
-d, --no-data Do not dump table data
-G, --triggers Dump triggers
-E, --events Dump events
-R, --routines Dump stored procedures and functions
These options are not enabled by default to keep backward compatibility with actual mixed solutions using Mysqldump for DDLs.
Locking reduce options
--trx-consistency-only      Transactional consistency only
You can think on this as --single-transaction for mysqldump, but still with binlog position. Obviously this position only applies to transactional tables (TokuDB included).  One of the advantages of using this option is that the global read lock is only held for the threads coordination, so it’s released as soon as the transactions are started.
GTIDs and Multisource Slave 
GTIDs are now recorded on the metadata file.  Also Mydumper is now able to detect a multisource slave (MariaDB 10.1.x) and will record all the slaves coordinates.
Myloader single database restore
Until now the only option was to copy the database files to a different directory and restore from it. However, we now have a new option available:
-s, --source-db                   Database to restore
It can be used also in combination with -B, --database to restore to a different database name.

As always, we have created a x86_64 RPM version for Centos 6.x in our repo:






Thursday, September 17, 2015

atrpms is dead and i need ffmpeg for CentOS 6

Berlin university seems to switch off the server running the atrpms repo. That is not official but after 10 days down it is easy to think it.

I have read some post in the CentOS list about this issue. In some answers,  people recommends to choose alternative repos. That is a crap. The recommendations are pointed to choose ffmpeg v0.10.x or similar.... oh come on! the last ffmpeg version today is 2.8.x ... is that serious?

In our infrastructure ffmpeg 2.2.x is valid for us, it is the atrmps version. So I have cloned, in the ENETRES Centos 6.x repo, the necessary packages (from atrmps) to install ffmpeg 2.2.x and mediainfo tool (with dependencies).

It you are in troubles, you can use our repo for that version install (note1: remember we have strong dependency from EPEL repo, note2: these packages are not official and they don't have any support).

Good luck! :)

Tuesday, September 8, 2015

Fast install on CentOS 6 for the old smokeping 2.6.8

As always i prefer the fast way....

When I was trying to install smokeping to make some tests to get latencies i found i had to compile it from sources and install some weird CPAN(perl) dependencies etc etc etc.... horrible to make a 5min-test.

So I "borrowed" some packages from here and from there..... and finally we have all the packages to make a "yum install smokeping" without problems.

This is the old 2.6.8 version so, mainly, forget ipv6 support.

As always,  for CentOS 6, at enetres repo.


Friday, June 26, 2015

varnish-vmod-geoip RPM package for Varnish > 4.0.1 CentOS 6

This Varnish module exports functions to look up GeoIP country codes and requires GeoIP library.

Module config and info here: https://github.com/varnish/libvmod-geoip

Updated!

:)

Monday, May 11, 2015

Ansible + Linode API + CentOS

Fast Mode ON! If you dont understand anything.... try to ask it in the comments.

Requirements for CentOS:
  • yum install pip
  • pip install linode-python
  • pip install chube

Template:


This values are from the API:

plan: 1           #cheapest 
datacenter: 6     #newmark NJ
distribution: 127 #centos 6.5

There are different values and you will need to ask them to the API so, to see full info of these three from Linode API (distributions IDs, datacenters and plans), you can run this nodejs script:


Dont forget the sudo npm install linode-api -g

:)

Fast Mode OFF!

Monday, April 27, 2015

Forcing ansible playbooks to concrete hosts (and vagrant version)

This is a fast workaround to force to run a playbook to a concrete host.
Important: You must to have the host added to the ansible host inventory.

You will need to convert hosts to a variable. From:

- name: Installing base server template
  hosts: all
  gather_facts: true
  roles:
   - base
   - ntpenabled


To:

- name: Installing base server template
  hosts: '{{ hosts }}'
  gather_facts: true
  roles:
   - base
   - ntpenabled

And now, in terminal for running the playbook:

ansible-playbook <playbook.yml> --extra-vars="hosts=<ip_or_hostname_here>"


and for vagrant:

  config.vm.define "test" do |test|
     test.vm.box = "chef/centos-6.6"
     test.vm.network "private_network", ip: "10.1.1.13"
     test.vm.provision "ansible" do |ansible|
       ansible.playbook = "ansible/playbooks/base.yml"
       ansible.sudo = true
       ansible.extra_vars = {
          hosts: "ip_or_hostname_here"
       }
     end
  end

Tuesday, April 21, 2015

Ansible + Vagrant: forget your interactive prompts (SOLVED)

If you have a playbook with something like this:

- name: Installing test box
  hosts: all   
  connection: paramiko
  vars_prompt:
     - name: "hosthname"
       hosthname: "Give me a hostname:"
       private: no
  gather_facts: true
  roles:
   - base
   - redisenabled
   - nodebase


 And you are trying to run it with vagrant following this Vagrantfile piece:

  config.vm.define "test" do |test|
     test.vm.box = "chef/centos-6.6"
     test.vm.network "private_network", ip: "10.1.1.13"
     test.vm.provision "ansible" do |ansible|
       ansible.playbook = "ansible/playbooks/test.yml"
       ansible.sudo = true
     end
  end

This var (hosthname) is not interactive in Vagrant, you never will be asked.

What is the trick? I tried this workaround and i liked it:

  • Just in case i would create a default value for the variable.
  • Force the value of the variable in the Vagrantfile

So, the final config files would be:
  • Playbook:
- name: Installing test box
  hosts: all   
  connection: paramiko
  vars_prompt:
     - name: "hosthname"
       hosthname: "Give me a hostname:"
       private: no
       default: "test01-default"
  gather_facts: true
  roles:
   - base
   - redisenabled
   - nodebase

  • Vagrantfile
  config.vm.define "test" do |test|
     test.vm.box = "chef/centos-6.6"
     test.vm.network "private_network", ip: "10.1.1.13"
     test.vm.provision "ansible" do |ansible|
       ansible.playbook = "ansible/playbooks/test.yml"
       ansible.sudo = true
       ansible.extra_vars = {
          hosthname: "test01"
       }
     end
  end



Wednesday, April 1, 2015

Boost C++ library RPM packages for CentOS 6

I have created some RPM packages from Boost C++ libraries, 1.54.0-8.20.2, 1.55.0,  1.56.0 1.57.0 1.58.0 and 1.59.0 for CentOS x64 (no 32bits sorry).

Building the Boost C++ Libraries with:

Performing configuration checks

    - 32-bit                   : no
    - 64-bit                   : yes
    - arm                      : no
    - mips1                    : no
    - power                    : no
    - sparc                    : no
    - x86                      : yes
    - lockfree boost::atomic_flag : yes
    - has_icu builds           : yes
warning: Graph library does not contain MPI-based parallel components.
note: to enable them, add "using mpi ;" to your user-config.jam
    - zlib                     : yes
    - iconv (libc)             : yes
    - icu                      : yes
    - compiler-supports-ssse3  : yes
    - compiler-supports-avx2   : no
    - gcc visibility           : yes
    - long double support      : yes
    - zlib                     : yes

Component configuration:

    - atomic                   : building
    - chrono                   : building
    - container                : building
    - context                  : building
    - coroutine                : building
    - date_time                : building
    - exception                : building
    - filesystem               : building
    - graph                    : building
    - graph_parallel           : building
    - iostreams                : building
    - locale                   : building
    - log                      : building
    - math                     : building
    - mpi                      : not building
    - program_options          : building
    - python                   : building
    - random                   : building
    - regex                    : building
    - serialization            : building
    - signals                  : building
    - system                   : building
    - test                     : building
    - thread                   : building
    - timer                    : building
    - wave                     : building


Easy to add:
sudo wget http://repo.enetres.net/enetres.repo -O /etc/yum.repos.d/enetres.repo
sudo yum install boost-devel

:)

Tuesday, March 3, 2015

ping to multiple hosts at the same time with fping

How to ping to multiple hosts with fping  showing what hosts are between 200ms to 999ms of latency to detect hosts with network issues in the LAN?:


IPVD.txt is a file with the IP list. No limits. I was using 400 IPs.
the "watch" tool to check the results periodically is cool.

:)

Wednesday, February 18, 2015

supervisord in CentOS 7 (systemd version)

Hello,

Fast installation in CentOS 7 for this "helper" to the queues service in laravel or django framework. EPEL package too old so:
  1. yum install python-setuptools python-pip
  2. pip install supervisor
  3. mkdir -p /etc/supervisord
  4. echo_supervisord_conf > /etc/supervisor.d/supervisord.conf
  5. forked systemd init script  (thx to Jiangge Zhang) in /usr/lib/systemd/system/supervisord.service:


And: 
  1. systemctl enable supervisord
  2. systemctl start supervisord

User=nginx is useful to run this process as nginx user. You can change it but the user must be in the system.

Monday, February 9, 2015

Nikto , sqlmap, Curl ... + avoiding CloudFlare challenge in CentOS6 in terminal (Solved)

It is possible you find with this situation.
  • No windows environment
  • Just text browser
  • You want to run a "nikto"
  • the target/host is protected with cloudflare.
Result: everything is false-positive:

+Server: cloudflare-nginx
+ Uncommon header 'cf-ray' found, with contents: 1aad22aaaaaaa7-MAD
+ Uncommon header 'x-frame-options' found, with contents: SAMEORIGIN
+ Cookie __cfduid created without the httponly flag
+ No CGI Directories found (use '-C all' to force check all possible dirs)
+ Server banner has changed from 'cloudflare-nginx' to '-nginx' which may suggest a WAF, load balancer or proxy is in place
+ "robots.txt" contains 1 entry which should be manually viewed.
+ lines
+ /crossdomain.xml contains 0 line which should be manually viewed for improper domains or wildcards.
+ Server leaks inodes via ETags, header found with file /favicon.ico, inode: 2221478, size: 1150, mtime: 0x4c35de66b2900
+ Uncommon header 'cf-cache-status' found, with contents: HIT
+ /kboard/: KBoard Forum 0.3.0 and prior have a security problem in forum_edit_post.php, forum_post.php and forum_reply.php
+ /lists/admin/: PHPList pre 2.6.4 contains a number of vulnerabilities including remote administrative access, harvesting user info and more. Default login to admin interface is admin/phplist
.
.
(a lot of more lines)

If you repeat it again with a "verbose" mode in other window like:

tcpdump -A -s0 port 80 |grep title

you will see:

 <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>
  <title>Just a moment...</title>


What is happening?

In some sites CloudFlare offers (for protection of the site) one challenge before the real webpage. There are two types:

  • Javascript challenge
  • Captcha challenge

The second option is the normal option when you are using Tor to visit the site. There is not a good solution for that option.



For the first one, the cloudflare-scrape project is our solution. You can develop whatever you want with that module for python.

For our problem, i was in CentOS, the procedure was:
  1. yum install python-requests #this is for installing the package dependencies but request library for centos is not enough, >= 2.x is a MUST 
  2. yum install python-pyV8 #look at my post about pyV8 RPM 
  3. yum install python-pip #to install the newest request module 
  4. pip install requests --upgrade #this is the correct module version
  5. yum install ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/halocaridina:/security/CentOS_CentOS-6/noarch/nikto-2.1.5-8.1.noarch.rpm #our nikto in centos
Later:

git clone https://github.com/Anorov/cloudflare-scrape

and I made a fast-ugly script (i am not developer) with the module:

import sys
import requests
import cfscrape

sess = requests.session()
sess.mount("http://", cfscrape.CloudflareAdapter())
sess.get (sys.argv[1])

print "\"cf_clearance\"=\"%s\";\"__cfduid\"=\"%s\"" % (sess.cookies["cf_clearance"] , sess.cookies["__cfduid"])


Now, the sugar: We have to use the same user agent in nikto and cloudflare-scrape. Both of them permit to change the user-agent.

Now we run the script:

python  myscriptcf.py http://www.xxxxxxxxxxx.com/
"cf_clearance"="4960843aaaaaaaaaaaaaaaaa42c4bdb23aaaaaaaaaaa-aaaaaaaaaaa3-1205";"__cfduid"="df59aaaaaaaaaaaaaaaaaaaa53"

This cookie goes to the STATIC-COOKIE in the /etc/nikto/config.

and now, retry-time: Re-run nikto and you try to look at the "verbose" screen with the output of tcpdump:

<title>HOLAHOLAWEBSITE</title>
<title>HOLAHOLAWEBSITE</title>
<title>404 Not Found</title>
<title>404 Not Found</title>
<title>404 Not Found</title>
<title>404 Not Found</title>
<title>404 Not Found</title>
<title>404 Not Found</title>
<title>404 Not Found</title>
<title>404 Not Found</title>
<title>404 Not Found</title>

Yeah, challenge accepted and it works! ;)

Other example:

SITE=www.sitewithcloudflarechallenge.com; curl -s -s $SITE -A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/34.0.1847.116 Chrome/34.0.1847.116 Safari/537.36' |grep title
 <title>Just a moment...</title>

Script (we need to clean the quotes here - different format to nikto config):

import sys
import requests
import cfscrape

sess = requests.session()
sess.mount("http://", cfscrape.CloudflareAdapter())
sess.get (sys.argv[1])

print "cf_clearance=%s;__cfduid=%s" % (sess.cookies["cf_clearance"] , sess.cookies["__cfduid"])

and...

SITE=www.sitewithcloudflarechallenge.com; curl --cookie `./myscriptcf.py $SITE` -s $SITE -A 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/34.0.1847.116 Chrome/34.0.1847.116 Safari/537.36' |grep title

<title>sitewithcloudflarechallenge</title>

Cool?. You think about to make a proxy with this now.... yeah! very coooooool!


Wednesday, February 4, 2015

FOSDEM 2015

F.O.S.D.E.M 2.0.1.5

very crazy!




:-O

Thanks to everybody.

Tuesday, January 27, 2015

Fast access to the history host table of Netcraft site from terminal

Fast and ugly but it works. Useful for detecting old IPs and system OS. Netcraft history host table from terminal:


Netcraft uses javascript so I chose casperjs for the scraper:

Monday, January 19, 2015

pyV8 RPM for CentOS 6

First RPM package for the project pyV8. You will save all the compiling process ;)
I used the last revision in the svn today (r586).

It depends of the boost library but i have RPMS for that in the repo ;)

There we go:

yum install python-pyV8

Loading mirror speeds from cached hostfile
 * base: mirror.trueinter.net
 * epel: mirror.uv.es
 * extras: mirror.trueinter.net
 * updates: mirror.trueinter.net
Resolviendo dependencias
--> Ejecutando prueba de transacción
---> Package python-pyV8.x86_64 0:1.0-preview_r586svn.el6 will be instalado
--> Procesando dependencias: libboost_python.so.1.55.0()(64bit) para el paquete: python-pyV8-1.0-preview_r586svn.el6.x86_64
--> Procesando dependencias: libboost_system.so.1.55.0()(64bit) para el paquete: python-pyV8-1.0-preview_r586svn.el6.x86_64
--> Procesando dependencias: libboost_thread.so.1.55.0()(64bit) para el paquete: python-pyV8-1.0-preview_r586svn.el6.x86_64
--> Ejecutando prueba de transacción
---> Package libboost_python1_55_0.x86_64 0:1.55.0-1 will be instalado
--> Procesando dependencias: boost-license1_55_0 para el paquete: libboost_python1_55_0-1.55.0-1.x86_64
---> Package libboost_system1_55_0.x86_64 0:1.55.0-1 will be instalado
---> Package libboost_thread1_55_0.x86_64 0:1.55.0-1 will be instalado
--> Ejecutando prueba de transacción
---> Package boost-license1_55_0.x86_64 0:1.55.0-1 will be instalado
--> Resolución de dependencias finalizada

Dependencias resueltas

======================================================================================================================
 Paquete                  Arquitectura
                                    Versión                      Repositorio                                    Tamaño
======================================================================================================================
Instalando:
 python-pyV8              x86_64    1.0-preview_r586svn.el6      enetres                                         10 M
Instalando para las dependencias:
 boost-license1_55_0      x86_64    1.55.0-1                     enetres                                         39 k
 libboost_python1_55_0    x86_64    1.55.0-1                     enetres                                        130 k
 libboost_system1_55_0    x86_64    1.55.0-1                     enetres                                         40 k
 libboost_thread1_55_0    x86_64    1.55.0-1                     enetres                                         62 k

Resumen de la transacción
======================================================================================================================
Instalar       5 Paquete(s)

Tamaño total: 11 M
Tamaño total de la descarga: 271 k
Tamaño instalado: 11 M
Está de acuerdo [s/N]:


As always, the pyV8 RPM package is in our repo: http://repo.enetres.net/

:)






Tuesday, January 13, 2015

nginx 1.7.6 RPM CentOS 6 + yaoweibin no_buffer patch + fancyindex

New nginx 1.7.6 RPM for CentOS 6 with fancyindex, yaoweibin no_buffer patch and all the modules:

nginx version: nginx/1.7.6
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) 
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-mail --with-mail_ssl_module --with-file-aio --with-ipv6 --add-module=/home/dag/rpmbuild/SOURCES/ngx-fancyindex --with-http_spdy_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'

Here: http://repo.enetres.net/repoview/nginx.html in the repo.

:)