Category "Tutorials"

Using Import Io To Extract Data Easily Example Usage Of Import Io Api

- - JavaScript, Tutorials, Web
Hey Guys, here in this post I will explain about one the most reliable and easy to use extraction api provider which is an open source tool named import.io

What is import.io?

Import.io is an open source, web-based data extraction platform which enables general users to access data from websites without writing any code. Import.io also serves API for extraction of necessary data which makes it very essential for programmers needing dynamic data from certain websites or data sources. Using import.io’s feature, data from websites can be queried easily either manually or with the use of an API. Using the service, the whole web can be considered as a big database for the machines and other applications to read the essential data which would definitely be more cumbersome while doing manually.

There can be numerous cases where a website does not provide API for data extraction. In cases like this, coding tactics to extract the right piece of information is possible but not very easy. Hence, import.io is the best fit because it provides API to extract the dataset more easily.

Example use of import.io’s API to extract data

1. Open up your browser and type in the following url to get to the coolest platform import.io . SignUp there, it’s free.

http://import.io

2. For this example, we are using a webpage of seasky.org. Below is the url where the dataset we want is present.

http://www.seasky.org/astronomy/astronomy-calendar-2015.html

3. Type in the url in the placeholder as shown below and press the Get Data button. Below is the similar screen-shot.

importdotiodataextract

4. Now we get the dataset we were looking for. Below is the screenshot of how it looks. At the buttom right corner is a button Get API. Press the button to get the API of import.io to extract the dataset. Now press copy this to my data.

importdotiogetdata

5. You now are able to query the dataset in various formats including JSON and TSV. Click on the Get API tab to see the structure of the url for query the dataset.

importdotiogetapi

6. At the end of the url is the parameter _apikey=YOUR API KEY. While using the url in your application you need to replace the YOUR API KEY with the alphanumeric api key provided by import.io. You can view this key by pressing on the parameter and typing your password for import.io account.

Using import.io API in code

Below is the piece of code I am using in my application to get the dynamic data from seasky.org

function getCalenderInfoFromServer(calenderYear){
    $.ajax({
        url: baseUrlDefault + calenderYear + baseUrlEnd + baseAPI,
        type: 'GET',
        dataType: 'json',
        success: function (data) {
            console.log(data);
            parseEventDetails(data.results);
        },
        error: function (e) {
            alert(e);
        }
    });

}

Thanks for reading.

How To Setup And Remotely Access A Linux Machine With Openssh Via Putty Exe

- - Applications, Python, Tutorials
Hello Readers. Here in this post, I will be explaining on how to setup the essentials accessing a device remotely from another device in some other network. Follow along .

To better understand the tutorial, read the background behind writing this article here

How to reserve an ip for your device?

I will be explaining on reserving an ip for a device via router’s settings. My router’s model is TL-WR740N, brand TP-Link. However the sole concept and procedure is similar for all the routers. First, go to your router’s settings,

192.168.0.1 (in my case)

Now provide valid credentials for user and password. For most of the routers, the default username and password is admin

Click on the DHCP option, present in the menu in left side in case of TP-Link

Now go ahead and click on Address Reservation

Click Add New so you can reserve an ip address for a device. You will be required to enter the Mac Address of the device.

Open up terminal with the combination of keys Ctrl + Alt + T in your linux machine and enter the following command.

ifconfig

Since I connect via WI-FI, I look for Hwaddr under wlan0.

Go ahead and paste this mac address to the address reservation menu.

After this, enter a desired ip address, in my case 192.168.0.106. Check enable and save the settings.

Once done, you will be required to reboot the router to save these settings. That’s all for a static local ip.

Setting up openssh-server and port Forwarding

It is dead simple to set up openssh server in linux machine. Follow along,

Enter the following command to get openssh-server

sudo apt-get install openssh-server

Now, to keep the openssh running, hit the following command

sudo service ssh start

The default port for ssh is 22

Another step is to forward the incoming traffic to my router’s public ip in port 22 to the local ip of the linux device. In my case to 192.168.0.106

For this, we again go to the router’s settings. Open up browser and go to 192.168.0.1 or something similar to open the router’s settings.

Find the menu named Forwarding or something similar and click the option. Now under the sub-menu click on virtual servers.

Once done click Add New

Fill the details :

Service Port : 22

Internal Port : 22

IP Address: 192.168.0.106 (the local ip of the machine with openssh running)

Protocol: All (TCP/UDP)

Status: Enabled

Go ahead and save it.

What have we done so far? Assigned a static ip to a device. Installed openssh-server in the linux machine and run it. Forwarded the incomming traffic to port 22 of the router to the local ip of the machine.

Another step. Based on your ISP, the public ip for your internet surfing may change but is static most of the time. In my case the ip barely changes. So all I have to do is rot the public ip which is what I did.

How to access a device remotely via ssh while on Windows machine

Prerequisite: The machine running openssh-server must be powered on. However it is not necessary to login. Also openssh-server remains in running state by default when device is powered on.

Download putty.exe from here. Putty is a telnet and ssh client for windows. The size of the software is very small and you need not install it. It’s a download and run software.

Once downloaded, run putty.exe. Enter the public ip of your home network, specify port 22 and check ssh then press open. This will open a terminal. If the connection was established meaning both the home network and the network you are at is connected to internet, you will be prompted for login credentials of your remote device. Providing the correct login credentials will give you access to your device at home network via command line.

Search for a file and send an email with attachment remotely via ssh and some python

Now that you have access to your linux machine remotely. You can use ls commands to find the location of the file you are looking for.

In my case, it’s present inside /home/bhishan/lispcognitive.odt

To send this file via email, we will be using python interactive shell.

Run the command to open python shell

python

We will be using two standard modules in python namely smtplib and email to send email with an attachment.

The snapshot shows the codes I used to send email

python codes to send email attachment

python codes to send email attachment

Below, I’ve nicely formatted the codes in a more readable way.

import smtplib
from email.mime.application import MIMEApplication
from email.mime.multipart import MIMEMultipart
from email.mine.text import MIMEText
from email.utils import formatdate

me = "fromemail@domain.com"
you = "toemail@domain.com"
password = "password"
msg = MIMEMultipart(
    From = me,
    To = you,
    Date = formatdate(localtime=True),
    Subject = "Sending email with attachment via python"
)
msg.attach(MIMEApplication(file("/home/bhishan/listcognitive.odt").read()))

s = smtplib.SMTP_SSL("smtp.gmail.com")
s.login(me, password)
s.sendmail(me, [you], msg.as_string())
s.quit()

If you have any questions regarding the post or the codes, comment below, so we can discuss.

Mining google search result using python – Making SEO kit with python

- - Applications, Python, Tutorials, Web

whatpageofsearchamion program in python

By the end of this read, you will be able to make a fully functional SEO kit where you can feed the keywords associated with certain url and the program will show the position of the url in google search. If you know some tools like “whatpageofsearchamion.com”, our program will be similar to that tool.

I was writing for a trekking based website and I had to make sure my writeup appears in a good position in google search. The keywords being competitive, I had to keep track of day to day report of the article’s status. Therefore I decided to write a simple program that would find my article’s position in google search. Continue Reading

Mining Facebook – Mining the Social Web using python

- - Python, Tutorials

Mining data from different sources has become a trend for the past few years. The need of structured and characteristic data has lead the data miners to advice their machines to mine social data. Twitter, Google + and Facebook are some of the social networks, now serving as a mountain of behavioral data. In this post Facebook will be our Zen source and python will be the miner. Python programming language is well known for it’s capability to withdraw web data effectively and efficiently. Cutting the long story short, today we will be using Facebook Graph API through python to mine the facebook page likes for each individual adjusting in our friend list. Continue Reading

Grab alexa rank and write to google spreadsheet using python

- - Tutorials

How to grab alexa rank using python?

By the end of this read you will be able to extract alexa rank and write it to a google spreadsheet programmatically. This can be helpful for seo experts as they have to constantly analyze the alexa ranking and submit it to his/her boss in a regular basis or for personal reference. Some times you have a huge number of websites to take care of, especially when your main expertise is search engine optimization. Going to alexa’s website and writing ranks for each website is a cumbersome task to deal with. The code provided below is in python programming language. Executing the code with valid gmail id, password, spreadsheet id, worksheet id associated with the spreadsheet you want to work with will append website link and current alexa rank with date to the spreadsheet. Don’t worry about spreadsheet id and worksheet id, I will let you know on how to get these items. Continue Reading

Make a 2048 game in python – 2048 game codes in python

- - Tutorials

2048 Game in Python

By the end of this read you will be able to make a popular game 2048 on your own. Python programming language will be used to implement the logic of the game in this tutorial.2048 is a game that reached maximum popularity in my college. This started with an email from my college’s director of academics with the header 2048 : A game to make. I choose python programming language to build 2048 game. Continue Reading

How to grab certain text from gmail using imaplib and json in python

- - Tutorials

Grab certain text from gmail using python

I tried this sometimes a month ago. This article could help on extracting parts of an email(GMAIL). As I walk further I’ll show you how to extract a single word(basically sounds funny). I’ll be using python programming language in order to grab a word programmatically.

The background behind this program will drive me to explain about a practice in my college. Everyday we receive an email in a formatted manner which contains a word, its meaning, normal day use and an example sentence accommodating this word. The idea of extracting the word from email and saving it somewhere in the local machine came from one of my classmate’s. Continue Reading

NPR API Tutorial II – How to use story API to parse a story having text, audio and image from npr.org

- - Tutorials

NPR API TUTORIAL

In this NPR API Tutorial, I’ll explain you on each line of codes to parse a complete story containing text, images and audio from npr.org using the NPR API while making queries to interface using python programming language. Have you got your npr api key? If not I’ve covered how to get it in my previous post which focuses on parsing a single text story of a specific category from npr.org Find it here. This post focuses on explaining each lines of code that we’ll use to parse a complete story containing text, audio as well as images from the Nation Public Radio’s official website. Continue Reading

NPR API Tutorial – How to use the NPR API to parse a story of any topics from npr.org website

- - Tutorials

NPR API Tutorial

NPR (National Public Radio) has been providing apikeys to public in order to attract people to code and build applications that uses NPR’s application programming interface. This is a tutorial on how to use the NPR api to parse a story of any topics from npr.org website. Python programming language would be my priority in this npr api tutorial. Continue Reading