Awadhesh Pratap Dwivedi &#8211; Teknonauts

Continue exploring at Teknonauts.com

#25 Python Programming – Setting up system P01

#26 Python Programming – Input-Output statements & Operators P02

#27 Python Programming – Mathematical Functions & List P03

#28 Python Programming – Conditional Statements and Dictionary

#29 Python Programming – Python Function, File Dictionary and Exception Handing

Awadhesh Pratap Dwivedi is an IT industry leader with over 13+ years of experience. He is excellent at providing an easy solution to complex business problems with his tremendous problem-solving skills. Currently, he is working with Oracle as a Principal Solution Engineer.

#29 Python Programming – Python Function, File Dictionary and Exception Handing

Awadhesh Pratap Dwivedi — Wed, 14 Jul 2021 07:45:49 +0000

Python Function

Python functions are classified into two types

Built-in functions (or) library functions ( example: print(),sort(),list() etc., )
User defined functions – which are created by user.

When you creating user-defined functions don’t use python keywords.

Definition

A function is a block of code (or) script .
A python function is defined as a group of statements (or) script placed inside another statement or scripts.
The main advantage of using function is its reusability in our working script.
Whenever we want to repeat a particular operation again and again, functions allow us to write the definition once and call function more than one time.

Function advantages

Provide reusable code /script, delivering enhanced modularity for your application.
Easy way to debug your script
Improve script scalability
Allows to import more than one external program
The Function operation block, previously mentioned as definition block ( action block ) will use def keyword
Function call is nothing but to invoke a function. In python function call has bottom-up approach.

Function definition syntax

def function_name() :

Function definition block

Syntax rules

Function name must begin with keyword def
Function name is case sensitive .Also avoid the usage of python keywords as a function name.
The function definition must end with colon :
All other lines inside a function, placed after the function definition are indented.

How to call a function ?

Let’s start with an example function hello().

def   hello() :
         print “This is hello function block”

Calling a function definition (or) invoking a function is nothing but writing a function name, followed by open and close parentheses ( ).

hello()  # This is simple function call
print  “Exit From script”

Whenever python interprets the line hello (), the hello function block will execute and after completing the execution, the control will return back to the script section

Example :

def   hello ( ) :
              print “This is hello function block”
print “B4 function call”
hello() # simple function call
print “After function call”

Output:-

B4 function call

This is hello function block

After function call

Function definitions can be placed in any order. But its execution is based on the order of the function calls

File : p1.py

import os    #  import  python os module file into local script
def    view( ):
          print “This is view function block”
          print “display list of mounted file system details :”
          os.system(“df –Th”)  # df –Th is a unix like operating system command
          print  “Exit from view function”
 
def   display() :
         print “This is display block”
         print “Current process details”
         os.system(“ps –f”)
         print “Exit from display function”
 
print   “Im Script Section “
display()  # 1st call  
view()  # 2nd call
print “Exit From script”

Nested function calls

Nested function calls are similar to nested structures where the function calls are placed inside another function.
These nested structures are applied in many situations which includes dependency-based actions and stack applications.
Let’s have an example

def  functionA() :
           print  “Im functionA”
           functionB() # nested call
 
def functionB():
          print “Im functionB”
          functionC()  # nested call
 
def functionC():
          print “Im functionC”
 
functionA()  # This is main script call
print “Exit from function block”

Output :-

Im functionA

Im functionB

Im functionC

Exit From function block

Function call with arguments

While passing arguments to a function, those arguments should be placed in the correct positional order.

def  hello(a1,a2) :  
          print “1st argument :”,a1
          print “2nd argument:”,a2
          print “Arguments types are :”
          print type(a1)
          print type(a2)


hello(10,20)                                       # arguments are numbers
hello(“/etc/passwd”, “/var/log/auth.log”)   # arguments are string
hello(“data1”,”data2”)                             # arguments are string
hello([‘one’,’two’,’three’],(“three”,”four”))         # arguments are list and tuple
hello({‘key1’:’Value1’,’key2’:’Value2’}, [10,20,30]) # arguments are dict and list

In python, the arguments passed inside a function are classified as

Required arguments
Keyword arguments
Default arguments
Variable-length arguments

Required arguments

These are the function arguments that have to be passed to a function in correct positional order.
The order of the arguments should match in both function call and function definition.
In case of any mismatch, the python return an error (Type Error) message and exits from the script without continuing to the next statement.

File : p1.py

print "I am python"
def a():  #  No arguments are given
          print "This is a block"
a(10,20)  #  call with two arguments
for v in [1,2,3,4,5]:
          print v
print "Exit from block"

Output :

I am python

Traceback (most recent call last):

File “a.py”, line 5, in

a(10,20)

TypeError: a() takes no arguments (2 given)

In the above example, the control stops executing the next looping statements placed after the function call.
Using exception handling technique we can handle this exception and can make the control move forward to the next statement of execution.
Here, the number of arguments (count) in the function call should also match exactly with the number of arguments in function definition.

def   display(a1,a2):

          print “This is display block”

          print “Received 1st argument:”,type(a1)

          print “Received 2nd argument:”,type(a2)

# The above code display block contains two arguments

display(“data1”,”data2”)  # valid call
display() # empty argument – invalid call
display(10,20,30) #more than two arguments - invalid call

Keyword arguments

Keyword arguments are related to the function calls.
When you use keyword arguments in a function call, the caller identifies the arguments by the parameter name.
This allows you to skip arguments or place them out of order because the Python interpreter is able to use the keywords provided to match the values with parameters.

def  display( name,age) :
          print  “Name:”, name
          print “Age:”,age
 
display(name=”Mr.Xerox”,age=55)

In python, the function will take the default value if the value is not passed in the function call as an argument.

def display(name=”Mr.Xerox”,dept=”Sales”) :
                   print “Name:”,name
                   print “Dept:”,dept
display()
display(name=”Mr.John”,dept=”Account”)
display(name=”Mr.Paul”)
display(dept=”Production”)

A default argument is an argument that assumes a default value if the value is not provided in the function call as argument.

Variable-length arguments

You may need to process a function for more arguments than you specify while defining the function.
These arguments are called variable-length arguments and are not named in the function definition, unlike required and default arguments.

def  functionname(*variable) :
                   function definition operation
 
functionname(arg1,arg2,arg3……argN)

Example

def display(a1,*a2):

print type(a1) # type ‘str’

print type(a2) # type `tuple`

display(“data”,10,20,30,40,50,60)

An asterisk (*) is placed before the variable name that holds the values of all non-keyword variable arguments.
These collection of non-keyword arguments form a tuple and it remains empty if no additional arguments are specified during the function call.

The *args vs **kwargs

We studied *args allows you to do is take in more arguments than the number of formal arguments that you previously defined.
With *args, any number of extra arguments can be treated as tuple type in formal parameters
The special syntax**kwargs in function definitions in python is used to pass a keyword variable-length argument list.
We use the name kwargs with the double star.
The reason is because the double star allows us to pass through keyword arguments (and any number of them).
A keyword argument is where you provide a name to the variable as you pass it into the function.
The kwargs as being a dictionary that maps each keyword to the value that we pass along it.
That’s the reason when we iterate over the kwargs there doesn’t seem to be ordered result.
kwargsreturn type will be ‘dict’ type.

>>> def    display(**kwargs):
...             print type(kwargs)

...

>>> display(name="Karthik",place="Bangalore")

 type 'dict'

>>> def    display(**kwargs):

 

...     for v in kwargs.keys():

...             print v,"\t",kwargs[v]

>>> display(name="Karthik",place="Bangalore")

#output
place   Bangalore

name    Karthik

Anonymous function

Anonymous function (or) lambda function i.e., functions unnamed or without name.
The general syntax of a lambda function:-

lambda argument_list : expression

The argument_ list consists of list of arguments separated by comma and the expression is
an arithmetic expression using these arguments.
You can assign the function to a variable to give it a name.

>>> f = lambda x, y : x + y

>>> f(10,20)

Scope of variables

All variables within a program may not be accessible at all locations in that program.
This depends on where you have declared that variable.
The scope of a variable determines the portion of the program where you can access a particular identifier.
There are two basic scopes of variables in Python ?
Global variables
Local variables
Variables that are defined inside a function is called local variables
Local variables can be accessed only inside the function, we can’t access outside the function

Example:

def    hello():
         Fname=”/var/log/result.log”   # local variable
         print “File name:”,Fname  
hello() # function call
print “Outside function  File name:”, Fname  ## NameError

We can use a global variable in other functions by declaring it as global in each function that assigns to it.

Global variables are accessible from anywhere in the script i.e., inside a function and outside a function.

Example :

def  hello():
         global Fname #  variable Fname is a global variable
         Fname=”/var/log/result.log”   
 
def  display():
       print “Result log file name is: “,Fname   # Result log file name is:/var/log/result.log
 
hello()
display()
print  “Outside function  File name:”  # display Outside function File name:/var/log/result.log

File Handling categories

Read a data from Keyboard(STDIN) to script write a data to FILE ( Not using Monitor(STDOUT) )
Read a data from FILE to Script display to Monitor ( Not using STDIN )
Read a data from one FILE to Script write a data to another FILE (Not using STDIN/STDOUT. (like: cp old new))
When we want to read from or write to a file we need to open it first.
When we are done, it needs to be closed, so that resources that are tied with the file are freed.
Hence, in Python, a file operation takes place in the following order.
- Open a file
- Read or write (perform operation)
- Close the file

How to open a file?

Python has a built-in function open() to open a file.
This function returns a file object, also called a handle, as it is used to read or modify the file accordingly.

Syntax :

Object=open(“Inputfile”)
>>> f = open("test.txt")    # open file in current directory
>>> f = open("/var/log/temp.log")  # specifying full path

We can specify the mode while opening a file.
In mode, we specify whether we want to read ‘r’, write ‘w’ or append ‘a’ to the file.
We also specify if we want to open the file in text mode or binary mode.
The default is reading in text mode. In this mode, we get strings when reading from the file.
On the other hand, binary mode returns bytes and this is the mode to be used when dealing with non-text files like image or exe files.

Python File Modes

Python File Modes
Mode	Description
‘r’	Open a file for reading. (default)
‘w’	Open a file for writing. Creates a new file if it does not exist or truncates the file if it exists.
‘x’	Open a file for exclusive creation. If the file already exists, the operation fails.
‘a’	Open for appending at the end of the file without truncating it. Creates a new file if it does not exist.
‘t’	Open in text mode. (default)
‘b’	Open in binary mode.
‘+’	Open a file for updating (reading and writing)

Example:-

f = open("test.txt")      # equivalent to 'r' or 'rt'
f = open("test.txt",'w')  # write in text mode
f = open("img.bmp",'r+b') # read and write in binary mode

How to close a file Using Python?

When we are done with operations to the file, we need to properly close the file.
Closing a file will free up the resources that were tied with the file and is done using Python close() method.
Python has a garbage collector to clean up unreferenced objects but, we must not rely on it to close the file.

f = open("test.txt)
# perform file operations
f.close()

How to read a file using python ?

Syntax :-

Object=Open(“Inputfile”,”mode”)
Example:-
>>> obj=open(“/var/log/result.log”,”r”)  # r – read mode by default.
>>> type(obj)

Read data from file in python we are using different functions.

1.read() – read the entire file contents and returns them as a single string
Example :-

cat IP1.txt
Data1
Data2
Data3
Data4
obj=open(“IP1.txt”)
obj.read()
‘Data1\nData2\nData3\nData4\n’
type(obj.read())

2. readline() – read the file contents line by line return as a single string

>>> obj=open(“IP1.txt”)
>>> obj.readline()
‘Data1’
>>> obj.readline()
‘Data2’
>>> obj.readline()
‘Data3’
>>> obj.readline()
‘Data4’
>>> obj.readline()
“ ”

3. readlines() – read the entire file contents and return as a single list type of data.

>>>obj=open(“IP1.txt”)
>>>type(obj.readlines())
>>>obj.readlines()
[‘Data1\n’,’Data2\n’,’Data3\n’,’Data4\n’]

4. next() – read line by line like readline() and return the next item from the iterator.

If default is given and the iterator is exhausted , control returns instead of raising StopIteration

>>>obj=open(“IP1.txt”)
>>>type(next(obj))
>>> next(obj)
‘Data1’
>>> next(obj)
 ‘Data2’
>>> next(obj)
 ‘Data3’
>>> next(obj)
 ‘Data4’
>>> next(obj)
StopIteration

Note that above object (obj) is iterable object. So will use for loop will read data from file.

>>> obj=open(“IP.txt”) # open a file , bydefault read mode
>>> for var in obj:  # iterable object
              print var.strip() # strip() function is used to remove \n char
>>> obj.close()  # close the file object
     Data1
     Data2
     Data3
     Data4
     Data5

This method is not entirely safe. If an exception occurs when we are performing some operation with the file, the code exits without closing the file.
A safer way is to use exception handling techniques which we will be discussing later.

try:
       f = open(“test.txt”)
except Exception as e:
     print e
else:
      for var in f:
               print var.strip
finally:
       f.close()

How to write to a File in Python?

In order to write into a file in Python, we need to open it in write ‘w’, append ‘a.

Syntax:-

open(“Filename”,” w”)

Example:-

f=open("test.txt",'w’):
f.write("my first file\n")
f.write("This file\n\n")
f.write("contains three lines\n")
f.close()

This program will create a new file named ‘test.txt’ if it does not exist. If it does exist, it is overwritten.
We must include the newline characters ourselves to distinguish different lines.

Using with keyword

with open("test.txt",'w') as f:
        f.write("my first file\n")
        f.write("This file\n\n")
        f.write("contains three lines\n")

This program will create a new file named ‘test.txt’ if it does not exist.
If it does exist, it is overwritten.
When using with keyword, file handler will be closed automatically.
No need to explicitly use f.close().

Using seek() method

We can change our current file cursor (position) using the seek() method.

>>> obj.seek(0)   # brings file cursor to initial position
0
>>>obj=open(“IP.txt”)
>>>obj.read()
‘Data1\nData2\nData3\nData4\nData5\n’
>>>obj.read()
“ “ # empty string
>>>obj.seek(0)  # brings file cursor to initial position
>>>obj.read()
‘Data1\nData2\nData3\nData4\nData5\n’

Python Directory Handling

A directory or folder is a collection of files and sub directories.
Using python os module we can handle the directories

Get Current Directory

We can get the present working directory using the getcwd()method.
This method returns the current working directory in the form of a string

>>> import os
>>> os.getcwd()
'C:\\Users\\Krosumlabs\\Desktop'

Changing Directory

We can change the current working directory using the chdir() method.

>>> os.chdir('C:\\Python33')
>>> print(os.getcwd())
C:\Python33

The new path that we want to change must be supplied as a string argument to this method.
We can use both forward slash (/) and the backward slash (\) to separate the path elements.

List Directories and Files

All files and sub directories inside a directory can be known using the listdir() method.

>>> os.listdir()
['DLLs',
'Doc',
'include',
'Lib',
'libs',
'LICENSE.txt',
'NEWS.txt',
'python.exe',
'pythonw.exe',
'README.txt',
'Scripts',
'tcl',
'Tools']

Making a New Directory

We can make a new directory using the mkdir() method.
This method takes in the path of the new directory. If the full path is not specified the new directory is created in the current working directory.

>>> os.mkdir('test')
>>> os.listdir()
['test']

Renaming a Directory or a File

The rename() method can rename a directory or a file.
The first argument is the old name and the new name must be supplied as the second argument.

>>> os.listdir()
['test']
>>> os.rename('test','new_one')
>>> os.listdir()
['new_one']

Removing Directory or File

A file can be removed (deleted) using the remove() method.
Similarly, the rmdir() method removes an empty directory.

>>> os.listdir()
['new_one', 'old.txt']
 
>>> os.remove('old.txt')
>>> os.listdir()
['new_one']
 
>>> os.rmdir('new_one')
>>> os.listdir()
[]

However, note that rmdir() method can only removes empty directories.
In order to remove a non-empty directory we can use the rmtree() method inside the shutil module.

>>> os.listdir()
['test']
>>> os.rmdir('test')
Traceback (most recent call last):
...
OSError: [WinError 145] The directory is not empty: 'test'
 
>>> import shutil
>>> shutil.rmtree('test')
>>> os.listdir()
[]

Multidimensional data structures

Combination of List,tuple and dictionary structures

List of List

>>> confs=[["p1.log","p2.log","p3.log"],["host01","host02"],
                          ["/etc",  ["passwd",”gpasswd"] ]  ]
>>> type(confs)
>>> confs[0]  # display 0th index
['p1.log', 'p2.log', 'p3.log']
>>> confs[1]  # display 1st index
['host01', 'host02']
>>> confs[2] # display 2nd index
['/etc', ['passwd', 'gpasswd']]
>>> confs[2][1] # list of list 2nd index of 1st index
['passwd', 'gpasswd']
>>>
>>> confs[1] # 1st index
['host01', 'host02']
>>> confs[2][1]
['passwd', 'gpasswd']
>>> confs[2][1][0] # list of list
'passwd'

List of tuple

# List inside tuples
>>> IPs=[('host01','host02','host03'),('host04',('host05'))]
>>> type(IPs)
>>> IPs[0]
('host01', 'host02', 'host03')
>>> type(IPs[0])
>>> IPs[-1]
('host04', 'host05')
>>> IPs[-1][0]
'host04'
>>> IPs[-1][1]
'host05'

Tuple of Tuple

# Read only mode
>>> files=("/etc/passwd",("p1.log","p2.log"),("p3.log"),("p4.log"))
>>>
>>> type(files)
>>>
>>> type(files[1])
>>> files[1]
('p1.log', 'p2.log')
>>>
>>> files[1][0]
'p1.log'

Tuple of List

>>> IPs=(['10.20.30.40','10.20.30.50','10.20.30.65'],['/etc/passwd','/etc/hosts','/etc/gpasswd'])  # tuple of list
>>> type(IPs)
>>> IPs[0]  # 0th index is List
['10.20.30.40', '10.20.30.50', '10.20.30.65']
>>> type(IPs[0]) # List type
>>> IPs[0][0] # tuple of list
'10.20.30.40'
>>> IPs[0][1]
'10.20.30.50'
>>> IPs[1]
['/etc/passwd', '/etc/hosts', '/etc/gpasswd']
>>> IPs[1][-1]
'/etc/gpasswd'
>>> IPs[-1]
['/etc/passwd', '/etc/hosts', '/etc/gpasswd']

Dictionary of List

# Single key points to multiple values ( list )
 Confs={'hosts':['/etc/hosts','/etc/hostname','/etc/sysconfig/network-scripts’]}
# ‘hosts’ is a key – value is list
>>> type(Confs)
>>> Confs['hosts']
['/etc/hosts', '/etc/hostname', '/etc/sysconfig/network-scripts']
>>> type(Confs['hosts'])
>>> Confs['hosts'][0]  # from dict key is pointing to 0th index list
'/etc/hosts'
>>> Confs['hosts'][-1] # from dict key is pointing to last index
'/etc/sysconfig/network-scripts'

Dictionary of tuple

# Dictionary key  value is tuple
 Confs={'hosts':('/etc/hosts','/etc/hostname','/etc/sysconfig/network-scripts’)}
>>> type(Confs)
 
>>> Confs['hosts'] # dictionary key is  tuple value
('/etc/hosts', '/etc/hostname', '/etc/sysconfig/network-scripts')
 
>>> type(Confs['hosts'])
>>> Confs['hosts'][0]  #  0th index
'/etc/hosts'
>>> Confs['hosts'][1]
'/etc/hostname'
>>> Confs['hosts'][2]
'/etc/sysconfig/network-scripts'

Dictionary of Dictionary

# Dictionary is holding another dictionary
>>> Confs={'host1':{'file1':'/etc/hosts'},'host2':{'file2':'/etc/hostname'},
                      'host3':{'file3':'/etc/sysconfig/network-scripts'}}
>>> type(Confs)
>>> type(Confs['host1'])  
>>> Confs['host1'] # host1 key value is dictionary
{'file1': '/etc/hosts'}
>>> Confs['host1']['file1']  # dictionary of another dictionary
'/etc/hosts'

Exception Handling in Python

Exceptions

An exception is a signal that an error or other unusual condition has occurred.
When that error occurs, Python generate an exception that can be handled, which avoids your program to crash.
Exceptions are convenient in many ways for handling errors
Below is some examples explaining some common exceptions errors in Python:

Examples

>>> var=10
>>> print Var  # Invalid variable –python case sensitive
Traceback (most recent call last):
  File "", line 1, in
NameError: name 'Var' is not defined
>>> a=['one','two','three']
>>> a[3] # invalid index
Traceback (most recent call last):
File "", line 1, in
IndexError: list index out of range
 
>>> import Http  # module is not loaded
Traceback (most recent call last):
  File "", line 1, in
ImportError: No module named Http
# Temp.log file is not exists
>>> open("Temp.log")
Traceback (most recent call last):
  File "", line 1, in
IOError: [Errno 2] No such file or directory: 'Temp.log'

Handling an exception

In basic terminology we are aware of try/except clause.
The code which can cause an exception to occur is put in the try block and the handling of the exception is implemented in the except block.

Syntax

try:

operation

except Exception :

Handle the exception

Example


NameError: name 'Var' is not defined
>>> try:
...     Var # Exception is occured here
... except NameError:
...     print "Invalid variable"
...
Invalid variable
>>>try:
…     print 1/0
…except ZeroDivisionError:
       print "You can't divide by zero."

The error handling is done through the use of exceptions that are caught in try blocks and handled in except blocks.
If an error is encountered, a try block code execution is stopped and transferred down to the except block.

Handling multiple exceptions

Syntax:-

try:
          operation
 except Exception1:
           Handle the exception
 except Exception2:
           Handle the exception
 except Exception3:
           Handle the exception
 .......
except Exception'N':
           Handle the exception
 
Example:-
>>> try:
...     open("Temp.log") # File is not available
... except NameError:
...     print "Invalid variable"
... except IOError:
...     print "File is not exists"
... except ZeroDivisonError:
...     print "Divide by zero"
...
File is not exists

Try … except … else clause

The else clause in a try/ except statement must follow all except clauses, and is useful for the code that must be executed if the try clause does not raise an exception.

try:
        operation
except Exception:
        handle_the_exception_error
else:
         # follows the code that there is no exception    
Example :-
>>> try :
...     var=10
... except NameError:
...     print "Name error"
... else:
...     print "No Exception"
...     print "var value is:",var
No Exception
var value is: 10

Try … finally clause

The finally clause is optional.
It is intended to define clean-up actions that must be executed under all circumstances.
Whether the exception occurs or not, the finally block will always execute.
Make sure that the else clause is run before the finally block.

Example:

File : e1.py

try:
           obj=open("Temp.log")
except IOError:
           print "Invalid file"
else:
           print "Read operation"
            for var in obj:
                     print var
                     obj.close
finally:
            print "-----Thank you----"

Let Temp.log file does not exist. So the exception occurs and the control moves to the except clause.
It displays ‘invalid file’ message and then will go to finally block displaying a ‘—-Thankyou —-‘.

C:\Users\Krosumlabs\Desktop>python e1.py

Invalid file

—–Thank you—-

Now create a Temp.log file and then run the Script discussed above.

Now, there is no exception. So the control goes to else clause, followed by the finally block.

C:\Users\Krosumlabs\Desktop>python e1.py

Read operation

dat1

data2

—–Thank you—-

A finally clause is always executed before leaving the try statement, irrespective of whether an exception has occurred or not.

Raising an Exception

In Python programming, exceptions are raised when corresponding errors occur at run time, but we can forcefully raise it using the keyword raise.
We can also optionally pass values to the exception to clarify why that exception was raised.

>>> raise KeyboardInterrupt
Traceback (most recent call last):
KeyboardInterrupt
>>> raise MemoryError("This is an argument")
Traceback (most recent call last):
MemoryError: This is an argument
>>> try:
...     a = int(input("Enter a positive integer: "))
...     if a <= 0:
...         raise ValueError("That is not a positive number!")
... except ValueError as ve:
...     print(ve)
...    
Enter a positive integer: -2
That is not a positive number!

Python Built-in Exceptions

Illegal operations can raise exceptions. There are plenty of built-in exceptions in Python that are raised when corresponding errors occur.

We can view all the built-in exceptions using the local() built-in functions as follows.

>>> locals()[‘__builtins__’]

This will return us a dictionary of built-in exceptions, functions and attributes.

Reference

Continue exploring at Teknonauts.com

#25 Python Programming – Setting up system P01

#26 Python Programming – Input-Output statements & Operators P02

#27 Python Programming – Mathematical Functions & List P03

#28 Python Programming – Conditional Statements and Dictionary

#28 Python Programming – Conditional Statements and Dictionary

Awadhesh Pratap Dwivedi — Thu, 24 Jun 2021 12:03:20 +0000

Python Conditional Statements

if statement

The if statement contains a logical expression using which data is compared and a decision is made based on the result of the comparison

if( condition-expression ):

Syntax :

if( condition-expression ):

Operation Block

If the Boolean expression evaluates to True, then the block of statement(s) inside the if statement is executed.
If Boolean expression evaluates to False, then the first set of code after the end of the if statement(s) is executed.
Note: The indentation is important in python. After conditional statement, on the next line followed by indent, write down true block operation

if..else style

An else statement can be combined with an if statement
If the condition is not True , the control will go to the else block part

Syntax :

if (condition):

True Block operation

else:

False Block operation

Note: After the else statement, the next line followed by the indentation is the False block operation

if..elif..elif..else style

Based on the outcome, if we need to determine which action to take place and which statements to execute, then the choice will be Multi conditional statements

Syntax:-

if ( condition1) :

True block -1

elif ( condition2):

True Block-2

elif (condition3):

True Block-3

…

elif (conditionN):

True Block-N

else:

False Block

Python will test the condition1 expression, if condition1 is True, the control will go to Trueblock -1, executes the python code then exits from the block
If condition1 fails, then the control will test the condition2 block
If condition2 is True, execute True Block-2 operation, then exit from control statement so on
Only if all the conditions are failed control will go to else block

Example :

if ( “AB” == “AB”) :

print (“Im True Block1”)

elif (100 >50):

print (“Im True Block2”)

elif (“SAB” != “sab”) :

print (“Im True Block3”)

else :

print (“False Statement”)

In the above example, all the given conditions are True
As per the flow, the Python will test the 1^st condition ( “AB” == “AB”)
Here condition1 is True, so the control will go to True block
In case if 1^st condition is False, the control will go to next condition

Result : “Im True Block1”

Looping in Python

A looping statement allows us to execute the code blocks more than one time
Python looping statements are executed in two ways
- List style – Execution of the block depends on the number of elements in the list
- Conditional style – – Execution of the block depends on the conditional statement

Python for loop

Syntax

for variable in list-of-elements:

Operation block

# Note:

Just like conditional statements, looping statements also needs proper indentation.

for, in are keywords.

If a list-of-elements in turn contains a list or a tuple or an unnamed expression list, the loop evaluates them first
And then, the first item in the list is assigned to the iterating variable
Next, the statements block is executed.
Further, each item in the list is assigned to iterating variable, and the statement(s) block is executed until the entire list is finished

Example 1

for var in "host01","host02","host03" :  
  print "Host name:",var       # Block will run 3 times

Result :-

Host name:host01

Host name:host02

Host name:host03

Example 2

IPs=['10.20.30.40','10.20.30.50','10.20.30.60','10.20.30.66','10.20.40.60']
for var in IPs:
      print "System IP:",var
      print "-------------------"

Result:-

System IP: 10.20.30.40

———————————–

System IP:10.20.30.50

———————————–

System IP:10.20.30.60

———————————–

System IP:10.20.30.66

———————————–

System IP:10.20.40.60

The above looping statement will run 5 times
List variable IPs is holding 5 elements so block will run 5 times

Example 3

# Display the CPU Load Balance for 5 times

import os
for v in 1,2,3,4,5:
         os.system("uptime")

Example 4

import os
for v in [1,2,3,4,5]: <======= List contains 5 elements
            os.system("uptime")

Example 5

import os
for v in (1,2,3,4,5):  <======= tuple contains 5 elements
              os.system("uptime")

Example 6

import os
for v in ('data1','data2','data3','data4','data5'):   <======= tuple contains 5 elements
              os.system("uptime")

All the above examples have same result.
It displays the current CPU load balance result 5 times

range()

By using the range() method, we can iterate over the looping operations, The range function takes number (any integer),say n as argument and returns a list of values start from 0 to n-1

Example:

range(5)  ==>  [0,1,2,3,4]
range(n,m)  ==> start from 'n' value to 'm-1' value
range(1,5)  ==> [1,2,3,4]
range(n,m,count)   ==>  start from 'n' value to 'm-1' for every count occurrence
range(1,7)   ==>  start from 1 to 6   ==> [ 1,2,3,4,5,6 ]   ==>  for every occurrence single increment
range(1,7,3)  ==> every 3 increment  ==> [1,4]

How to use range() in for loop statements ?

Example:

for v in range(5) :   # [0,1,2,3,4]  No.of elements in the list is 5 ,so block will run 5 times
           print "Hello..",v

Result:-

Hello..0

Hello..1

Hello..2

Hello..3

Hello..4

for v in range(1,10,3): 
          print "value of v:",v  # Block will run 3 times

Result :-

value of v:1

value of v:4

value of v:7

Using else Statement with Loops

Python supports the usage of an else statement with a loop statement
In case the else statement is used with a for loop, the else statement is executed when the loop has finished iterating the list
Generally else statements are optional in looping statements and are usually used to write a footer message
else block can be used with both for and while loops

Example:-

for v in range(5):
         print "Hello-->",v
else:
         print "Thank you"

Result:-

Hello–>0

Hello–>1

Hello–>2

Hello–>3

Hello–>4

Thank you

Python While loop

A while loop in Python continuously executes a statement as long as the given condition is true.

syntax

While (condition ) :

Operation –block

The Loop will test the condition. If condition is True, the operation block is executed and again the condition is tested and if condition is True again, it repeat the operation
This goes on until the condition fails where the loop terminates
Here the operation block execution is determined only by the condition and not by the list, tuple or list of values

Example:-

count=0
while(count<5):
  print "Count value :",count
  count=count+1

Result :

Count value : 0

Count value : 1

Count value : 2

Count value : 3

Count value : 4

While with else statement

Example :-

count=0
while(count<5):
  print "Count value :",count
  count=count+1
else :
   print "Thank you"

Result

Count value : 0

Count value : 1

Count value : 2

Count value : 3

Count value : 4

Thank you

Looping statements

1) break – Terminate from loops

The most common use of break is when some external condition is triggered requiring a hasty exit from a loop
The break statement can be used in both while and for loops

Example

while True:

Fname=raw_input( “Enter a first name:”):
if( Fname == “Ram” ):
          print “Matched”
          break  # Exit from while loop

The above loop is an un-conditional ( always true ) infinite loop
Once the condition is matched, block will get terminate

2) Continue-continues on the next iteration of the loop

It returns the control to the beginning of the while loop
The continue statement rejects all the remaining statements in the current iteration of the loop and moves the control back to the top of the loop
The continue statement can be used in both while and for loops

Example :

for  var in [‘Mon’,’Tue’,’Wed’,’Thu’,’Fri’]:
if(var == “Wed”):
      continue # Skip the wed, continue to the next statement
else:
      print var

Result:

Mon

Tue

Thu

Fri

3) Pass statement

The pass statement is a null operation; nothing happens when it executes
The pass is also useful in places where your code will eventually go, but has not been written yet (e.g., in stubs for example) ?

Example:–

for letter in 'Python':
       if letter == 'h':
             pass
             print 'This is pass block'
             print 'Current Letter :', letter
             print "Good bye!"

Result:-

Current Letter : P

Current Letter : y

Current Letter : t

This is pass block

Current Letter : h

Current Letter : o

Current Letter : n

Dictionary

Python dictionary is an unordered collection of data items
Like list python dictionary is also modifiable
A dictionary contains collection of items, which is of (key,value) pair

Syntax :-

Dictionaryname={‘Key’ : ‘Value’ }

Keys are unique string whereas value can duplicate
Each key is separated from its value by a colon (:), the items are separated by commas, and the whole thing is enclosed in curly braces
The values of a dictionary can be of any type, but the keys must be of an immutable data type such as strings, numbers, or tuples

How to create a dictionary?

Creating a dictionary is as simple as placing items inside curly braces { } separated by comma

Syntax :-

Dictionaryname={‘Key1’ : ‘Value1’, ‘Key2’: ‘Value2’…….’Key N’: ‘Value N’ }

Examples:-

Hosts={‘ip’ : ’10.20.30.40’, ‘file’:’/etc/hosts’ , ‘port’ : 22 }

Hosts is the name of the dictionary

ip is a key & its value is an IP address (10.20.30.40 )

file is a key & its value is /etc/hosts

port is a key & its value is 22

Generally, dictionaries have a key and the corresponding value expressed as a pair, key: value

How to access the elements from a dictionary?

Similar to list, in order to get the data from a dictionary, we use index [ ] notation

Using index [ ] :

dict_name[‘key’]

Keys can also be used with the get() method to access the elements from a dictionary.

Using get() :

dict_name.get(‘key’)

Examples:-

emp={‘name’ : ‘Mr.Kris’ , ‘ID’ : ‘E123’, ‘dept’:’sales’}
print “Emp name is:”, emp[‘name’]
print emp[‘name’],” Working department is:”,emp.get(‘dept’)
Mr.Kris Working department is: sales

Size of a dictionary

We use len() function to display the size of a dictionary

emp={‘name’ : ‘Mr.Vishnu’ , ‘ID’ : ‘E123’, ‘dept’:’sales’}
print len(K) 
3

Data inside a dictionary are unordered elements
In order to get list of keys from dictionary, we use keys() function

Syntax:-

dictionary_name.keys()

keys() returns a collection of values and its return type will be of list type.

K=emp.keys()

# Determine keys() return type

type(K)

(or)

type(emp.keys())

Dictionary Manipulation

Dictionary are mutable, which means that we can add new data (key:value) into an existing dictionary or we can update/modify the value of an existing dictionary using an assignment operator

Adding new data to a dictionary

Whenever adding a new data to an existing dictionary, we should define a new unique key and a value

Syntax :-

dictionary_name[‘Key’]=”Value”

Examples:-

>>>emp={‘name’ : ‘Mr.Vishnu’ , ‘ID’ : ‘E123’, ‘dept’:’sales’}
>>>print  len(emp.keys())
3
>>>emp[‘phone’]=”+91 9901233456”
>>>emp[‘place’]=”Bangalore”
>>>print  len(emp.keys())
5

Modifing an existing data in a dictionary

Syntax:-

dictionary_name[‘Key’]=”Updated Value”

Example:-

>>>emp={‘name’ : ‘Mr.Vishnu’ , ‘ID’ : ‘E123’, ‘dept’:’sales’, ’phone’ : ’+91 9901233456’, ‘place’ : ‘Bangalore’}
>>>print “Before Modification : “, emp[‘dept’]
>>>emp[‘dept’]=”PRODUCTION” # modify existing value
>>>print “After Modification : ”,emp[‘dept’]

Result:-

After Modification : PRODUCTION

Before Modification : sales

Delete a single record from dictionary

Syntax :- del(dictionary_name[‘Key’])

To clear the entire contents of a dictionary

Syntax:-

dictionary_name.clear( )

Example:-

>>>emp={‘name’ : ‘Mr.Vishnu’ , ‘ID’ : ‘E123’, ‘dept’:’sales’}
>>>print “Total no.of records :”,len(emp.keys())  
Total no.of records : 3
>>>del(emp[‘dept’] ) # to delete a single record
>>>print “Total no.of records :”,len(emp.keys())
Total no.of records : 2
>>>emp.clear( ) # remove all entries in  emp dictionary 
>>>print “Total no.of records :”,len(emp.keys())
Total no.of records : 0

To delete an entire dictionary

del(emp)

Dictionary methods and examples

Syntax

dictionary_name.clear() #removes all elements of dictionary

>>>emp={‘name’ : ‘Mr.Vishnu’ , ‘ID’ : ‘E123’, ‘dept’:’sales’}
>>>emp.clear()  - Now the size of dictionary is zero

Syntax : dictionary_name.setdefault(key,default=None)

The method setdefault() is similar to get(), but the difference is that it sets dictionary_name[key]=default if the key does not exist in the dictionary
Similar to get() method, it returns the value corresponding to the key
In case if the given key is not available; it returns the provided default value

emp={‘name’:’Vishnu’,’dept’:’sales’}
emp.setdefault(‘name’)  # displays name (key) value as Vishnu.
#If passed as invalid key as an argument , it will return as None.
emp.setdefault(‘Name’) -> None 
if(emp.setdefault(‘Name’)==None):
             print “Invalid Key”

The above program input key (‘Name’) is an invalid key, so the default value (None) will be returned
Here the condition is True .So the control goes to True block and the script say ‘Invalid key’

if(emp.setdefault(‘name’) ==None):
                 print “Invalid key”
else:
                 print “value of name is :”,emp[‘name’]

In the above code, the given key is a valid key (‘name’). Hence it returns a string(“Vishnu”) and not ‘None’.

value of name is: Vishnu

Syntax

dictionary.update(dict2)

The method update() adds the dictionary dict2’s key-values pairs in to dict
This function does not return anything
But all items of dict2 is added into dict1

Items={‘code’:’P-123’,’cost’:1000.54}
Files={‘fname’:’Invoice.txt’, ‘size’:’100M’}
Items.update(Files) # Files dictionary is added into Items dictionary 
>>> Items
{‘code’:’P-123’,’fname’:’Invoice.txt’,’cost’:1000.54,’size’:’100M’}
# Note that dictionary are unordered data type
>>> print “Total no.of records :”,len(Items.keys())
Total no.of records:4

Syntax

dictionary_name.keys()

This method returns a list of all keys available in a given dictionary

>>>emp={‘name’ : ‘Mr.Vishnu’ , ‘ID’ : ‘E123’, ‘dept’:’sales’}
>>>emp.keys()
[‘name’, ’ID’, ’dept’ ]

The return type of keys() will be always list type of data

Syntax

dictionary_name.items()

The method items() returns a list of dictionary’s (key, value) tuple pairs

>>>  emp={‘name’ : ‘Mr.Vinu’ , ‘ID’ : ‘E123’, ‘dept’:’sales’}
>>> emp.items()
[('dept', 'sales'), ('name', 'Mr.Vinu'), ('ID', ‘E123')]

Syntax:

dictionary_name.has_key(Key)

This method has_key() returns True if a given key is available in the dictionary, otherwise it returns a False.

>>>  emp={‘name’ : ‘Mr.Vinu’ , ‘ID’ : ‘E123’, ‘dept’:’sales’}
>>> emp.has_key(‘name’)
True
>>> emp.has_key(‘NAME’)  # invalid input key
False

Since dictionary_name.has_key(Key) returns a Boolean values (True / False ), it is mostly used in testing or validation

if(emp.has_key(‘name’)):  # input key is valid return will be True
               print “Name is Found:”,emp[‘name’]
else:
                print “Invalid key”

Since the condition satisfies, the above code displays ‘Name is Found: Mr.Vinu’

if(emp.has_key(‘NAME’)):  # input key is invalid return will be False
          print “Valid Key”
else:
          print “Invalid Key”

Here the function returns false and hence it displays ‘Invalid Key’.

Other dictionary methods

pop() method

The pop() method removes and returns an element from a dictionary having the given key.

syntax :

dictionary.pop(key[, default])

Similar to deleting a single key del(dict[‘key’])

emp={‘name’ : ‘Mr.Vinu’ , ‘ID’ : ‘E123’, ‘dept’:’sales’}
>>>  emp.pop(‘dept’)  # same as  del (emp[‘dept’])
sales
>>> emp
{‘ID’:’E123’,’name’:’Mr.Vinu’}

pop( ) Parameters

The pop() method takes two parameters:
key – key which is to be searched for removal
default – value which is to be returned when the key is not in the dictionary
Return value from pop()
The pop() method returns:
If key is found – remove/pop the element from the dictionary
If key is not found – value specified as the second argument (default)
If key is not found and default argument is not specified – KeyError exception is raised

emp={‘name’ : ‘Mr.Vishnu’ , ‘ID’ : ‘E123’, ‘dept’:’sales’}
>>> emp.pop(‘ID’)
‘E123’
>>>emp.pop(‘post’,’manager’)
‘manager’
>>>emp.pop(‘K1’)  # invalid key
KeyError: ‘K1’

popitem()

The popitem() returns and removes an arbitrary element (key, value) pair from the dictionary.

syntax:-

dict.popitem()

The popitem() doesn’t take any parameters

Return Value from popitem()

returns an arbitrary element (key, value) pair from the dictionary.
When you run the program, the output will be

>>>emp={‘name’ : ‘Mr.Vishnu’ , ‘ID’ : ‘E123’, ‘dept’:’sales’}
>>>emp.popitem()
(‘ID’,’E123’)  
>>>emp
{‘dept’ : ‘sales’, ‘name’ : ‘Mr.Vishnu’ }
>>>emp.popitem()
(‘dept’:’sales’)
>>>emp
{‘name’ : ‘Mr.Vishnu’}
>>>emp.popitem()
(‘name’ : ‘Mr.vishnu’)
>>> emp
{}  # empty dictionary
>>> emp.popitems()  
KeyError : ‘popitem(): dictionary is empty’

Reference

Continue exploring at Teknonauts.com

#25 Python Programming – Setting up system P01

#26 Python Programming – Input-Output statements & Operators P02

#27 Python Programming – Mathematical Functions & List P03

#27 Python Programming – Mathematical Functions & List P03

Awadhesh Pratap Dwivedi — Fri, 18 Jun 2021 02:37:53 +0000

Mathematical functions

Python supports a wide variety of mathematical functions.

Function	Returns	Example
abs(x)	Returns the absolute value of x	x = -35;x = abs(x) print(x)
cmp(x,y)	Returns -1 if x < y> Returns 0 if x equals to y Returns 1 if x > y.	x = 6; y = 4 print( cmp(x,y) )
exp(x)	Returns the exponential of x	import math x = 6; print( math.exp(x) )
log(x)	The natural logarithm of x	import math x = 6; print( math.log(x) )
log10(x)	The base-10 logarithm of x	import math x = 6; print( math.log10(x) )
pow(x,y)	The result of x**y	import math x = 6; print( math.pow(x,2) )
sqrt(x)	The square root of x	import math x = 6; print( math.sqrt(x) )

List

Python List is a collection of data like array in Perl, shell script & ruby
Here data are not strictly of same type. It’s not like C – array
Data in side list are ordered elements (index based)
List are mutable i.e., we can modify the content of the list
Lists have methods that allow you to manipulate the values inside them.
Python list elements are separated by comma
List elements are enclosed by square brackets [ ]

How to create a list?

Syntax:-

listname= [data1, data2, data3, data4,data5….dataN ]

db=[ “oracle” , “sql”,”mysql”,”DB2”,”sqlite”]

|__ list name

[“oracle” , “sql” ,”mysql”,”DB2”,”sqlite” ]

0 1 2 3 4 index

# List index starts at 0

To determine list type using type function

>>>  db=[ “oracle” , “sql”,”mysql”,”DB2”,”sqlite”]
>>> type(db)

Accessing list values

=> List of elements accessed using the list name

>>>print db   
#output
[ “oracle” , “sql”,”mysql”,”DB2”,”sqlite”]

Access single data from list

Syntax:-

listname [index number ]

Example:-

# print 0^th index data

>>>print db[0]
#output
'oracle'
# print 1st index data
>>>print db[1]
'sql'

5	-4	-3	-2	-1	Index From Right to Left
[‘Oracle’,	‘sql’,	‘mysql’,	‘DB2’,	‘sqlite’]
0	1	2	3	4	Index from Left to Right

Slice Operator

db=[ “oracle” , “sql”,”mysql”,”DB2”,”sqlite”]
print db[1:3]  #output [ ‘sql’,  ‘ mysql’ ]
print db[2:]    #output  [‘mysql’,’db2’,’sqllite’ ]
print db[:2]   #output   [‘oracle',’sql`]

Create employee list

>>>emp=[ ‘E123’, ‘Mr.Visa’, ‘sales’,’10.20.30.40.50’,’12995.46’ ]
>>>print emp
[ ‘E123’, ‘Mr.Visa’, ‘sales’,’10.20.30.40.50’,’12995.46’ ]
>>>print  “Hello ”,emp [1]
Hello  Mr.Visa
>>>print  emp[1] ,” working department is: ”,emp[2]
 # Mr.Visa working department is sales

Update list values

# Modify sales department in to PRODUCTION

# we can modify list values through index

Syntax :-

listname[index]= “updated value”

>>>emp[2]=”PRODUCTION”  
>>>print   emp[1] ,”Current working department is:”,emp[2]
Mr.Visa Current working department is PRODUCTION

Find the size of list

Use len() function to display the total number of elements in the list.

>>>a1=[ 10,20,’data’,’/etc/passwd’,’10.20.30.40’ ]
>>>print  len(a1)  
5

Re-cap len() we used in string to count the number of characters

>>>print  len(“Ab”)
2

Delete List Elements

To remove a list element, you can use the del() function

Syntax:- del(listname)

Example:-

>>>a1=[ 10,20,’data’,’/etc/passwd’,’10.20.30.40’ ]
>>>print  a1
[ 10,20,’data’,’/etc/passwd’,’10.20.30.40’ ]
>>>print len(a1)
 5
>>>del(a1) # delete a list name
>>>a2= [ ‘data1’,’data2’,’data3’ ]
           #    0th         1st         2nd    index
>>>print  “Total no.of elements :”, len(a2)  
Total no.of elements: 3
>>>del(a2[1])  # delete 1st index data
>>>print  “Total no.of elements :”, len(a2)
Total no.of elements: 2

Basic List operations

List respond to the + and * operators much like strings; + Operator is used for list concatenation and * is used for list repetition operators .

>>>L1=[‘data1’,’data2’,’data3’]
>>>L2=[10, 20, 30]
>>>L1+ L2
[ ‘data1’, ’data2’, ’data3’ ,10 ,20 ,30 ]
>>>L1*3
[‘data1’,’data2’,’data3’, ‘data1’, ’data2’,’data3’, ‘data1’, ’data2’,’data3’]

Membership operators with list

>>>L1=[‘data1’,’data2’,’data3’]
>>> ‘data1’  in  L1   # test  data1 is placed in the given list (L1) 
True
>>> ‘dataX’  in  L1  # test dataX is placed in the given list( L1) 
False
>>> ‘data1’ not in L1
False
>>> ‘dataX’ not in L1  # test dataX is not placed in the given list
True

List Based Method

Syntax:- sorted(list) # displays sorted order of elements

>>>sh=[”expect”,”csh”,”tcsh”,”bash”]
>>>sorted(sh)  
bash ,csh ,expect,tcsh

Syntax:-

cmp(list1,list2) #compares corresponding elements of two lists.

list1 – This is the first list to be compared.

list2 – This is the second list to be compared.

If elements are of the same type, perform the compare and return the result.
If elements are different types, check to see if they are numbers.
If numbers, perform numeric coercion if necessary and compare.
If either element is a number, then the other element is “larger” (numbers are “smallest”).
Otherwise, types are sorted alphabetically by name.

l1=[‘a’,’b’]
l2=[‘a’,’b’]
l3=[‘A’,’b’]
cmp(l1,l2)  ==> 0
cmp(l1,l3) ==> 1
cmp(l3,l1)   ==> -1

Syntax:-

len(list) #returns the number of items in list

list – list for which number of elements is to be counted.

l1=[‘data1’,’data2’,’data3’]
print  “Total no.of elements:”,len(l1)  ==> Total no.of elements: 3

Syntax:-

max(list) #returns the elements from the list with maximum value.

L1=[10,20,330,500,25]
print max(L1) ==> 500

Syntax:-

min(list) # returns the elements from the list with minimum value.

L1=[10,20,330,500,25]
print min(L1) ==> 10

List methods

To get list of methods type help() function with argument as a list

>>> help(list)

The method append() appends a passed data into the existing list.

Syntax:-

list.append (data)

>>>sh=[‘/bin/sh’, ‘/bin/bash’,’/usr/bin/tcsh’]
>>>sh.append(‘/usr/bin/perl’)  # adding “/usr/bin/perl”  to  existing list

This method does not return any value but updates existing list.

>>> sh=[‘/bin/sh’, ‘/bin/bash’,’/usr/bin/tcsh’]

>>> sh.append(‘/usr/bin/perl’)  # adding “/usr/bin/perl”  to  existing list
>>> sh
[ ‘/bin/sh’, ‘/bin/bash’,’/usr/bin/tcsh’,’/usr/bin/perl’ ]

The method count() returns count of how many times data occurs in list.

Syntax:-

List.count(‘data’)

>>>L1=[‘unix’,’linux’,’unix’,’aix’,’linux’,’winx’]
>>>print L1.count(‘linux’)  
2
>>>print L1.count(‘winx’)
1

The method extend() appends the contents of seq to list

Syntax :-

List.extend(seq) where Seq is collection of elements

L1=[‘unix’,’linux’,’aix’]
L2=[‘data1’,’data2’]
L1.extend(L2)
print “Extended List :”, L1
Extended List :[ ‘unix’,’linux’,’aix’,’data1’,’data2’]
L3=[‘unix’,’linux’,’aix’]
print “Extend list:”, L3.extend(“ABC”)
Extend list: [‘unix’,’linux’,’aix’,’A’,’B’,’C’]

The method index() returns the lowest index in list where data appears

Syntax:-

List.index(data)

This method returns index of the found object otherwise raises an exception indicating that value does not found.

>>>L1=[‘unix’,’linux’,’aix’]
>>>print L1.index(‘linux’)
1

The method insert() inserts object data into list at offset index.

Syntax:-

List.insert(index,data)

index – This is the Index where the object obj need to be inserted.

data – This is the data to be inserted into the given list

This method does not return any value but it inserts the given element at the given index.

L1=[‘unix’,’linux’,’aix’]
L1.insert(2,”SOLARIS”)
>>> L1
[‘unix’,’linux’,’SOLARIS’,’aix’]

The method pop() removes and returns last data from the list.

Syntax:-

List.pop()
List.pop(data)
=>   This method returns the removed object from the list.
L1=[‘unix’,’linux’,’aix’,’solaris’,’winx’]
>>> L1.pop()
‘winx’
>>> L1.pop(2)
‘aix’

This method remove() does not return any value but removes the given object from the list.

Syntax:-

List.remove(data)
L1=[‘unix’,’linux’,’aix’,’solaris’,’winx’]
L1.remove(‘linux’) # remove linux element from L1 list

The method reverse() does not return any value but reverse the given object in the list.

Syntax:- List.reverse()


>>> L1=[‘unix’,’linux’,’aix’,’solaris’,’winx’]
>>> L1.reverse()
>>> L1
[‘winx’,’solaris’,’aix’,’linux’,’unix’]

The method sort() sorts objects of the list

Syntax:-

List.sort()

This method does not return any value but it changes the original list
Compare sorted() function returns the list of data in sorted order but keeps the original values unchanged

>>>L1.sort()
>>>L1
[‘aix’,’linux’,’solaris’,’unix’,’winx’]

Tuple

Python tuple is a collection of data like array in Perl, shell script, ruby
Here data are not strictly of same type. It’s not like C – array
Data in side list are ordered elements (index based) like list
Unlike List, Tuple is a sequence of immutable elements, means that tuple elements cannot be updated or modified.
Like List, tuple have methods that allow you only to perform manipulation on the values inside them.
Python tuple elements are separated by comma
Tuple elements are enclosed by parentheses

How to create a tuple?

Syntax:-

tuplename= ( data1, data2, data3, data4,data5….dataN  )
db=(  “oracle” , “sql”,”mysql”,”DB2”,”sqlite” )
|__ tuple name
>>> type(db)

Print elements of tuple

>>> print db
(“oracle” , “sql”, “mysql”, “DB2”,”sqlite”)

Accessing Single data from the Tuple

Syntax:-

Tuplename[index]

>> db=(“oracle” , “sql”, “mysql”, “DB2”,”sqlite”)
#print  1st index data from tuple
print db[1]  => like list 'sql'
print db[2:]  => from 2nd index to list of all  (‘mysql’,’DB2’,’sqlite’)

Basic Tuple operation

Syntax:-

tuplename.count (Value)

# return integer total number of occurrence of a given value.
db=(‘oracle’,’sql’,’oralce’,’mysql’)
db.count(‘oracle’)  ==>  2

Syntax:-

len(tuplename)

db=(“oracle” , “sql”, “mysql”, “DB2”,”sqlite”)
print “Total no.of DB’s :”,len(db)  
# Total no.of DB’s : 5

Removing individual tuple elements is not possible.
append () or insert() is not used in tuple.
We can’t modify single data from tuple
Like List, Tuple supports min() , max() cmp() functions.
tuple(seq) – Converts a list into tuple.
list(seq) –Converts a tuple into list

Advantages of Tuple over List

Since, tuples are quite similar to lists; both of them are used in similar situations as well.
However, there are certain advantages of implementing a tuple over a list.
Below listed are some of the main advantages
1. Since tuple are immutable, iterating through tuple is faster than with list. So there is a slight performance boost.
2. Tuples that contain immutable elements can be used as key for a dictionary. With list, this is not possible.
3. If you have data that doesn’t change, implementing it as tuple will guarantee that it remains write-protected.

Reference

Continue exploring at Teknonauts.com

#26 Python Programming – Input-Output statements & Operators P02

#25 Python Programming – Setting up system P01

#26 Python Programming – Input-Output statements & Operators P02

Awadhesh Pratap Dwivedi — Sat, 05 Jun 2021 16:34:00 +0000

Till now you know the basics of Python programming and able to make simple program. Now lets go little more deeper and understand Input & Output statements

Input and Output statement

The usage of input statement depends on the Python version being used.

Python 3 . x (Input)

The input() function is used to get input from STDIN which usually returns String type of data.
In the case of numerical input, perform an additional step of typecasting it using int() or float() function.

Syntax

var=input(“Prompt”)

Example :-

>>> x = input (“Enter X value : “)
Enter X value : 12
>>>x
‘12’ # Note the quotes enclosing the value referring string type data
>>>type(x)
>>>x = int ( input (“Enter X value : “))
Enter X value : 10
>>>x
10 # Numerical data-displayed without quotes
>>>type(x)

Python 2. x (Input)

An older version of Python where in addition to the input () method, we have raw_input() to get input from STDIN.
The input() in Python 2. x is slightly different from that in Python 3. x.
Here the input() returns the numerical type of data. Otherwise, syntax and usage are the same.
raw_input() returns string type of data.

Syntax

var=raw_input(“Prompt”)
Example
>>>y = input ( “Enter y : ”)
Enter y : 12
>>>type(y)
>>>x= raw_input (“Enter x : “)
Enter x : hai
>>x
‘hai’
>>>type(x)

Print statement

The usual way of displaying text to the STDOUT is print method. This is very similar to the printf statement in C language.
The usage of format specifier (%) in print method is similar to C style coding.

>>> print(“s”  %(“Hai”))
Hai  # we tell format specifier that 10 characters are expected.
 # Right formatting with white space prefixed.
>>>  print(“% -10s”  %(“Hai”))
Hai       #Left Formatting with trailing white space & width=10

Let us make it clearer with another print statement.

>>>  print(“% -10s”  %(“Hai”)), ; print(“All”)
#output 
Hai       All

Usage of comma(,) after a print() method suppresses the newline character.
Recall semicolon(;) used to concatenate multiple statement on single line.
Note the above way of comma usage works only in Python 2.x versions & not in Python 3.x.

format( ) method

The another style of using print() method along with the usage of format().

>>>  x=3; y=5; print ( “The Total of {} and {} is {}”.format(x,y,x+y))
#output
The Total of 3 and 5 is 8

Using sys module

Other way of displaying text to the STDOUT is by using sys module

In Python 3.x,

>>>import sys
>>>sys.stdout.write(“Ever Inspiring”)
Ever Inspiring14
>>>sys.stdout.write(“Dr.A.P.J.Abdul Kalam”)
Dr.A.P.J.Abdul Kalam20

Here, the write() method returns the text followed by its total length and a newline character.

In Python 2.x,

>>>import sys
>>>sys.stdout.write(“Ever Inspiring”)
Ever Inspiring
>>>sys.stdout.write(“Dr.A.P.J.Abdul Kalam”)
Dr.A.P.J.Abdul Kalam

Here the write() method in sys module works similar to print() except that it does not allow the trailing newline character.

Python operators

Arithmetic operators

Arithmetic operators
Python supports arithmetic operations like addition (+), multiplication (*), division (/), subtractions (–), modulus( %), exponent (**) and floor division (//).

Example:-

>>>x = 30
>>>y = 80
>>>total = x + y
>>>print(total)
110
>>>2**3
8
>>>20//3
6
>>>-20/3
-7

Relational operators

The following are some relational operators used to perform comparisons.
==, != (or) <>, <, >, <=, >= . These operators return Boolean values ( True / False ).

>>>a= int( input(“Enter A : ”))
Enter A : 12
>>>b= int( input(“Enter B : ”))
Enter B : 3
>>>a != b
True
>>>a <> b
True
>>>a == b
False

Logical operators

Logical operators includes and, or, not

>>> (a != b) and (a>b )
True
>>>not(a <> b)
False

Bitwise operators

Performs bit by bit operation

Consider A=5, B=10

A= 0000 0101 (5)

B= 0000 1010 (10)

A&B = 0000 0000 (0) # copies bit if bit in both the operands are set.

A|B = 0000 1111 (15) # copies bit if bit in any of the operands are set.

~A = 1111 1010 (10) # one’s complement

A<<2> (20) # Binary left shift

A>>2 = 0000 0001 (1) # Binary right shift

Python Identity Operators

Identity operators are used to compare the objects, not if they are equal, but if they are actually the same object, with the same memory location:

is –> Returns True if both variables are the same object . Example : x is y

is not–> Returns True if both variables are not the same object. Example : x is not y

Python Membership Operators

Membership operators are used to test if a sequence is presented in an object:

in –> Returns True if a sequence with the specified value is present in the object. Example : x in y

not in –> Returns True if a sequence with the specified value is not present in the object. Example : x not in y

Reference

Continue exploring at Teknonauts.com

#25 Python Programming – Setting up system P01

#25 Python Programming – Setting up system P01

Awadhesh Pratap Dwivedi — Fri, 04 Jun 2021 21:34:00 +0000

If you are just starting your journey as a Programmer, then I highly recommend you to start with Python. Here are some reason-

Why you should Learn Python

First of all, Python is an easy-to-learn, interpreted programming language.
Furthermore, it has efficient high-level data structures, which allow you to write complex operations in fewer statements than in C, C++, or Java.
Object-oriented programming is a lot easier than in languages like Java.
Python has become one of the most popular programming languages among developers and programmers because of its Interactive and Extendable features. They praise it for its clean syntax and code readability.
Python is a general-purpose, high-level programming language.
Python is both object-oriented and imperative and it can be even used in a functional style as well.
Python programs are portable, i.e. they can be ported to other operating systems like Windows, Linux, Unix, and Mac OS X, and they can be run on Java and .NET virtual machines.
Emerging technologies like AI, ML, and Data Science are mostly developed using Python

A Brief History of Python

Invented in the Netherlands, the early 90s by Guido van Rossum
Named after Monty Python
Open-sourced from the beginning
Considered a scripting language, but is much more
Scalable, object-oriented and functional from the beginning
Used by Google from the beginning
Increasingly popular

Python’s Benevolent Dictator For Life

“Python is an experiment in how much freedom program-mers need. Too much freedom and nobody can read another’s code; too little and expressive-ness is endangered.” – Guido van Rossum

Guido van Rossum

Advantages of Python

It’s surprisingly easy to embed Python, or better the Python interpreter into C programs.
The Python Standard Library contains an enormous number of useful modules and is part of every standard python installation.
We can write python program in object-oriented and procedure style
Python is a dynamic type language
Provides interface to all major databases
Automatic Garbage collection
We can execute python in different ways

Get Started with Python

Download and install python from – https://www.python.org/

Python IDEs

Komodo	Windows/Linux/Mac OS X	Multi-language IDE with support for Python 2.x and Python 3. Available as Komodo IDE (commercial).
PyDev	Eclipse	Free, open-source plugin for Eclipse — Allows Python, Jython, and IronPython editing, code-completion, debugger, refactoring, quick navigation, templates, code analysis, unittest integration, Django integration, etc.

Other popularly used IDEs are Notepad++, Spyder, Jupyter Notebook, Pycharm and Atom

Using the Python Interpreter

With the Python interactive interpreter, it is easy to check Python commands
The Python interpreter can be invoked by typing the command “python” without any parameter followed by the “return” key at the shell prompt.

Python Prompt

Python Basic

We use the Python interpreter as a simple calculator by typing an arithmetic expression

root@krosumlabs~]#  python {Enter-key}
>>> 12+2
37
>>> 2.40+0.25
2.65
>>> “AB”  + “SAB”    # string concatenation
ABSAB
>>> 5*3    # multiplication
15
>>>”AB”*3   # string repetition   
ABABAB
>>> 12.0 / 7 # floating point division
1.7142857142857142
>>>”AB” + 12 #Error- mixed types

Python follows the usual order of operation in the expression
The standard order of operations is expressed in the following enumeration
- Exponents and roots
- Multiplication and division
- Addition and subtraction

This means that we don’t need parenthesis in the expression :3 + (2 * 4)

>>> 3 + 2 * 4
11

Python provides numerous built-in functions (BIFs). Of which help () provides us with the help document on the related object.

>>>help (3)
# Help on int object……
>>>help (int)
# Help on int object……
>>>help (str)
# Help on string object……

Some other commonly used BIFs are type (), max () and dir ().

>>>type (12)
>>>type(‘12’) #data within quotes are considered to be string .
>>>max(12, 3.145678, 0.2344)
12
>>>dir (10)       #returns a list of data & functions associated with the object passed, here an integer (10).

Usage of the semicolon as a statement terminator is optional in python. However, it can be used to concatenate multiple statements in a single line

>>> 12+3; ‘Learning Well’; 33+7.222
15
‘Learning Well’
40.222

Python Comment Lines

Single Line Comments

Typically, you just use the # (pound) sign to comment out everything that follows it on that line.

>>> print("Not a comment")
       Not a comment
>>> # print("Am a comment")  # one line comment
>>>  import  os  #  loaded os module
>>>  os.system("uptime")  #  using system function display cpu load balance

Multiple Line Comments

Multiple line comments are slightly different
Simply use triple quotes before and after the part that we want to comment
It can be either triple single quotes (”’…….”’) or triple double quotes (“““…..”””)

'''  print("We are in a comment line")
     print ("We are still in a comment line")
'''
print("We are out of the comment")

Execute a Python script

Now Let us save our program in a source file
To save and edit programs in a file, we need an editor
There are lots of editors, but you should choose one, which supports syntax highlighting and indentation
Under Linux, you can use vi, vim, emacs, geany, gedit and umpteen others. So, after you have chosen your editor, you can input your mini script and save it as filename.py
The suffix .py is not really necessary under Linux but it’s good style to use it
But it is essential, if you want to write modules

root@krosumlabs~]#   vi  ab.py  
print  “Hello this 1st python code”
print   “There are many data structures used in python”
# this line single line comment
# python won’t read this line
‘‘‘
This is multiline comment
’’’
print  “End of the script”

( Save the program, exit from editor  ESC  :wq )

root@krosumlabs~]#   python ab.py # execute python file
Hello this 1st python code
There are many data structures used in python
End of the script

Structuring with Indentation

Python uses a different principle
Programs get structured through indentation; this means that code blocks are defined by their indentation
All statements with the same distance to the right belong to the same block of code,i.e. the statements within a block line up vertically
The block ends at a line less indented or the end of the file
If a block has to be more deeply nested, it is simply indented further to the right

Reference

Continue exploring at Teknonauts.com

#26 Python Programming – Input-Output statements & Operators P02

#23 Seven Best Practices for Enterprise Application Development

Awadhesh Pratap Dwivedi — Tue, 25 May 2021 06:54:34 +0000

Developing a new application is the art of your thinking and creativity. You not only focus on your business but research a lot about how your customer will respond. That is the reason we have experienced a very good focus on customer experience these and days. Here are some of the best practices that will help to develop an application for an Enterprise.

UI best practices

Provide a familiar look and feel: Standardizing the look and feel of software allows users to transfer the skills they learn on one piece of software to another. Training costs are minimized
Provide consistency: Standardization may occur in varying scopes. Examples include the various components of an application, applications that will be used together, and an operating system and all the software that runs on it. The more broadly a standard can be applied, the greater the benefits
Use human factors findings: Standards take advantage of the large body of human factors research and accepted practice. The authors of the standards assimilate and interpret the research, turning it into guidelines (“best practice”) for designers to follow
Streamline development: Standards make many design decisions routine. This frees designers to spend time on decisions that are more difficult or critical
Evaluate usability: Standards provide one basis for judging the usability of products. All else being equal, a product that meets an HCI standard should be more usable than one that does not
Comply with requirements: Standards compliance for the software may be required by the buyer (like Country specific rules, law etc.)

Follow User Interface Design Guidelines: 10 Rules of Thumb for more details

Single Sign-On best practices

All Web Portals and Mobile App would require secured access through sign in. The user profile would be maintained in the user session and would be used by all other applications. Here are some best practices to be followed

Choose the right product as an Identity Provider (IDP)
Verify that the identity directory is accurate
Secure all the components of the SSO system
Consider user privileges: Provide authorized access to each module

Learn more about how to implement security and single sign-on with your application – https://www.onelogin.com/learn/how-single-sign-on-works

Omni-Channel best practices

All Web Portals and Mobile App to be made compatible to defined service delivery channels. In a multi-channel environment, the user has access to a variety of communication options that aren’t necessarily synchronized or connected. However, during an omni-channel experience, there aren’t only multiple channels, but the channels are connected so you can move between them seamlessly. Best practices to be followed are –

Provide Real-Time Updates across all channels
Digitally Supplement In-Person Experiences: Not only should online interactions inform in-person experiences, but the two should blend for new, unique experiences
Leverage Messaging to Meet Audience Needs: More people than ever are using messaging to communicate with businesses, such as Facebook Messenger, so build your integration across these messaging platform

Loosely Coupled and Bounded Context Applications and modules

All IT application and their modules should be developed such that the features and functionality are made available as loosely coupled, self-contained, standards based and configurable services.

The Application functional dispositions are designed aligned to the loosely coupled architecture principle with business service supported by ministry specific applications connected to the integration platform. The common shared or supporting services would follow the common application principle. Following best practices may be followed to achieve decoupling.

Decouple at the module level
Decouple at the object level
Decouple at the processing level

Common Applications For Common/Shared Business Services

Construct and agree on common business supporting services. Develop application to support those services or enhance existing IT Systems to support the services. Best practice for developing shared application may be decided based on the need and functions of the enterprise.

Low Code/No Codding Service Platform

A low code/No Code development platform provides an environment to create services through graphical user interfaces and configuration instead of traditional computer programming. Promoting such best practices for application development may reduce lot of cost and time to the market.

Integrated Service Delivery Platform

An integrated service delivery platform, that forms the basic building blocks of domain specific services that would be built and re-used across the enterprise to develop all services

Conclusion

There are many best practices for designing the architecture for your application, better implementation, coding, and testing. There are best practices to automate your DevOps and another manual task during development and application rollout. But when you start thinking about an Enterprise as a whole, remember these 7 best practices to drive your application development and provide a unique experience to your customers.

Explore more at Teknonauts.com

#20 Oracle Database 21C World’s best and new age RDBMS

Awadhesh Pratap Dwivedi — Wed, 12 May 2021 11:33:21 +0000

World’s most popular Oracle database has come up with major upgrade in its database 21c. With this launch RDBMS is back in the game and challenging the use cases where NoSQL was supposed to be the best fit. New age RDBMS Oracle database 21c contains more than 200 new innovations including in-database machine learning AutoML, immutable blockchain table, graph processing optimization, in-memory optimization, JavaScript support, dramatic improvement to JSON operations, sharding, multitenant, security and many more.

Oracle database 21c is available on Oracle cloud and can be deployed as a Database Service Virtual Machine (for clusters and single instance) or Bare Metal service (single instance). It is also available with Oracle Cloud always free tier of Oracle Autonomous database with limited storage and CPU capacity. In addition, Oracle APEX (Application Express) Application Development is also available with Oracle database which is a new low-code service for developing and deploying data-driven enterprise applications quickly and easily.

Data-Driven Future

Oracle has consistently worked on managing and storing data in converged database which is more efficient and productive than breaking up into multiple single-use engines. Converged database addresses the data integrity, consistency and security issues. Now, let’s understand what is Converged database? A converged database is a multi-model, multitenant, multi-workload database. It supports the data model and access method each development team wants, without unneeded functionality getting in the way. It provides both the consolidation and isolation these different teams want but don’t want to think about. And it excels in all the workloads (like OLTP, analytics, and IoT) these teams require. Oracle Database 19c is the world’s first converged database.

Figure 1 Converge Oracle Database

Oracle database 21c has put more innovation to Oracle’s converged database and offers: best of breed support for all data types (e.g. relational, JSON, XML, spatial, graph, OLAP, etc.), and industry-leading performance, scalability, availability and security for all their operational, analytical and other mixed workloads. Oracle’s converged strategy also ensures that developers benefit from all Oracle Database 21c key capabilities (e.g. ACID transactions, read consistency, parallel scans and DML, online backups, etc.) – freeing them to focus on developing applications without having to worry about data persistence.

Figure 2 Multi-model innovation – Oracle Database

What’s new in Oracle Database 21c

This latest Innovation release introduces several new features and enhancements that further extend database use cases, improves developer, analyst and data scientist productivity, and increases query performance. Listed below is a subset of what’s new in Oracle Database 21c.

New Blockchain Tables

Blockchain as a technology has promised much in terms of solving many of the problems associated with the verification of transactions. While considerable progress has been made in bringing this technology to the enterprise, several problems exist. Arguably, the largest being the complex nature of building applications that can support a distributed ledger. Oracle Database 21c addresses this problem with the introduction of Blockchain Tables. These tables operate like any normal heap table, but with a few important differences. The most notable of these being that rows are cryptographically hashed as they are inserted into the table, ensuring that the row can no longer be changed later.

Figure 3 Blockchain Table – Oracle Database

New Native JSON Datatype

Oracle Database has had JSON support since version 12c and storing JSON data as a VARCHAR2 or a LOB (CLOB or BLOB). In Oracle Database 21c, JSON support is further enhanced by offering a native data type, “JSON”. This means that instead of having to parse JSON on read or update operations, the parse only happens on an insert and the JSON is then held in an internal binary format which makes access much faster. This can result in read and update operations being 4 or 5 times faster and updates to very large JSON documents being 20 to 30 times faster.

Figure 4 Native JSON Datatype

Figure 5 function JSON_TRANSFORM

JSON Performance in Oracle Database

Here is comparison of Oracle database performance with Mongodb

New JavaScript Execution inside the Database

Oracle Database 21c adds support for JavaScript via the Oracle Multilingual Engine (MLE), powered by GraalVM. JavaScript is a ubiquitous scripting language that, among its many uses, enables richer user interaction in web applications and mobile apps. It’s one of the few languages that runs in a web browser and can be used to develop both client-side and server-side code. There is a large collection of existing JavaScript libraries for implementing complex programs, and JavaScript works in conjunction with popular development technologies such as JSON and REST. The MLE automatically maps JavaScript data types to Oracle Database data types and vice versa. The JavaScript code itself can execute PL/SQL (stored procedures) and SQL through a built-in JavaScript module.

Figure 6 JavaScript Execution in Oracle Database

More Innovations to Support Multi-Workload

Figure 7 Multi-workload – Oracle Database

In-Memory improvement

Database In-Memory requires a lot of manual effort from DBAs, developers and users. Now you can set INMEMORY_AUTOMATIC_LEVEL to HIGH, and all columns will be considered for in-memory analysis.

Also, in-memory hash joins have been optimized using SIMD vectorization, resulting in up to 10x higher speed. Also, columnar scans have been sped up for the case where not every column is in memory. Now, the optimizer can perform a hybrid scan and fetch projected column values from the row store if needed, instead of needing to perform the scan entirely on the row store.

Figure 8 In-Memory Column Store – Oracle Database

Persistent Memory Store

“Dramatic performance improvement for I/O-bound workloads”

Oracle Database 21c added support for a Persistent Memory (PMEM) store on single instance databases. This allows for storage of data and redo in local PMEM on commodity hardware. Oracle Database 21c runs SQL directly on data stored in the mapped PMEM file system, eliminating the need for a large buffer cache and reduces the code path. And it provides much faster transaction durability and near-instant recovery. This feature is especially useful for workloads that tend to be I/O bound.

Figure 9 Persistent Memory Store – Oracle Database

AutoML

Oracle Database 21c makes it even simpler for data scientists and analysts to take advantage of in-database machine learning by providing Python machine learning interfaces to the database. This new client tool complements existing R and SQL interfaces already available. AutoML simplifies the development of predictive machine learning models by automating the model selection, feature selection, and parameter tuning processes required for building accurate models.

“Automatically build and compare Machine Learning models”

Figure 10 AutoML

More Innovations to Support Multi-Tenant

Figure 11 Multi-Tenant

An Oracle database is a multi-tenant container database (CDB) that holds one system seed pluggable database and any number of user-created pluggable databases (PDBs). (See the third diagram below.) Users interact only with the PDBs, and a user may never realize that the PDB they are using isn’t a standalone database. In the past, Oracle also supported non-CDB database instances, but that option is no longer supported in Oracle Database 21c.

Multi-tenant Enhancement

Figure 12 Data Guard Per-PDB Physical Standby

DbNest

DbNest provides Enhanced Security Isolation for both Container and Pluggable Databases. CDBs and PDBs each reside within their own security realms called a “Nest”, which are enforced using advanced O/S features. DbNest provides security isolation of following

Processes
CPU
Memory
Network
File access

Figure 13 DbNest

Conclusion

Oracle Database 21c is World’s best-converged database that comes up with 200+ new features for all data-driven application use cases. Please explore more exciting features ( https://docs.oracle.com/en/database/oracle/oracle-database/21/whats-new.html ) listed here and experience the new age RDBMS.

Oracle Database 21c More Features

Reference

https://oracle.com/

Explore more at Teknonauts.com

#19 Microservices Data Management: 7 Important Design Patterns

Awadhesh Pratap Dwivedi — Sun, 25 Apr 2021 06:24:29 +0000

What are microservices

Microservices – also known as the microservice architecture – is an architectural style that structures an application as a collection of loosely coupled services, which implement business capabilities. The microservice architecture enables the continuous delivery/deployment of large, complex applications. It also enables an organization to evolve its technology stack.

Need of design patterns for Microservices Data Management

We have decomposed the application in Microservices but Business demand integrated data that’s where we miss monolithic architecture. Since we have addressed some of the major challenges with microservices we introduced some powerful design patterns to address the data management need in microservices architecture.

Design Patterns for Data management

Database Per Service

When we have segregated the responsibilities at the application layer we want to maintain this decoupled nature till the database layer. So, what database architecture we should follow?

Forces

Services must be loosely coupled so that they can be developed, deployed and scaled independently
Some business use cases are not independent, they must interact with multiple services either to fetch the data or updated the data owned by other services
Some business use cases need to join the data owned by multiple services
Databases must sometimes be replicated and sharded in order to scale
Every microservice has different data storage requirement. For some services, RDBMS is the best option and for some NOSQL is the right fit

Solution

“Keep each microservice’s persistent data private to that service and accessible only via its API. A service’s transactions only involve its database”

Figure 1 Microservices – Database per Service

In the above diagram Data, A will be private to Service A and will not never be accessed directly by any service. There are many ways to keep a service’s persistent data private. You not necessarily need to set up a separate database service for all microservices. But you can use below for instance if you are using RDBMS

Private-tables-per-service – each service owns a set of tables that must only be accessed by that service
Schema-per-service – each service has a database schema that’s private to that service
Database-server-per-service – each service has it’s own database server.

Benefits

Services are loosely coupled
Each services are free to use best suited database for their need

Drawbacks

Implementing business transactions that span multiple services is not straightforward
Implementing queries that join data that is now in multiple databases is challenging
Complexity of managing multiple SQL and NoSQL databases

Shared Database

There are use cases when each business cases are so interlinked that you will end up touching most of Microservices in a single call. For systems, we must use a shared database.

Forces

Services must be loosely coupled so that they can be developed, deployed and scaled independently
Many use cases require data owned by multiple services
Use cases need to update data owned by multiple services

Solution

Use a (single) database that is shared by multiple services. Each service freely accesses data owned by other services using local ACID transactions.

Figure 2 Microservices – Shared database

Benefits

Easy for developer to create a query for required data
Easy to operate single set of data for ETL & other data management activities

Drawbacks

Database became monolithic and we have coupled the responsibilities at data layer
Reduced freedom to use best fit database for each microservices
Manage the schema switch at run time will slow down system

Saga

As we talked about business cases where we require to get the data owned by multiple services also there are transactions that need to update the data owned by multiple microservices. This pattern will address the problem related to transactions that span across multiple microservices.

Forces

Services must be loosely coupled so that they can be developed, deployed and scaled independently
Use cases need to update data owned by multiple services

Solution

Each business transaction that span across multiple services should be implemented as Saga. Saga is basically a sequence of local transactions where each transaction updates the database and trigger a event or message to execute the next transaction. In this series we also implement the fallback mechanism linked to each transaction so that if local transaction fails saga can execute sequence of compensating transaction to rollback the changes made by earlier transactions.

Figure 3 Microservices – Saga

There are two ways of coordination sagas:

Choreography – each local transaction publishes domain events that trigger local transactions in other services
Orchestration – an orchestrator (object) tells the participants what local transactions to execute

Example – Choreography-based saga

Figure 4: Microservices – Saga – Choreography

Example – Orchestration-based saga

Figure 5 Microservices – Saga – Orchestration

Benefits

It enables an application to maintain data consistency across multiple services without using distributed transactions
Resilient Architecture with high performance

Drawbacks

The programming model is more complex
In order to be reliable, a service must atomically update its database and publish a message/event

API Composition

API composition is designed to address the problem that comes as an outcome of implementing a Database per service pattern for Microservices. When we implement a Database per service pattern, we can no longer write queries that join data from multiple services. API composition will provide ways to implement queries in Microservices.

Forces

Services must be loosely coupled so that they can be developed, deployed and scaled independently
Address the use cases requesting data from multiple data sources

Solution

Implement a query by defining an API Composer, which invoking the services that own the data and performs an in-memory join of the results.

Figure 6 Microservices – API Composition

Benefits

It a simple way to query data in a microservice architecture

Drawbacks

Some queries would result in inefficient, in-memory joins of large datasets.

Command Query Responsibility Segregation (CQRS)

Command Query Responsibility Segregation pattern is also designed to address the issues after implementing Database per service pattern. it is no longer straightforward to implement queries that join data from multiple services. Also, if you have applied the Event sourcing pattern then the data is no longer easily queried. CQRS provide another mechanism to implement query hat retrieves data from multiple services in a microservice architecture.

Forces

Services must be loosely coupled so that they can be developed, deployed and scaled independently
Address the use cases requesting data from multiple data sources

Solution

In this approach, we define a view database which is the read-only replica of the actual data. This database is designed to support the query that require data from multiple data sources. CQRS define the application responsibility to keep the data up-to-date by subscribing to event published by service that own the data.

Figure 7 Microservices – CQRS

Benefits

Supports multiple denormalized views that are scalable and performant
Improved separation of concerns = simpler command and query models
Necessary in an event sourced architecture

Drawbacks

Increased complexity
Potential code duplication
Replication lag/eventually consistent views

Domain Event

A service often needs to publish data/events when it updates its database. These events might be needed, for example, to update a CQRS view database. Alternatively, the service might participate in a choreography-based saga, which uses events for coordination. Domain Event pattern provide a method for a service to publish an event

Forces

Services must be loosely coupled so that they can be developed, deployed and scaled independently
Address the use cases requesting data from multiple data sources

Solution

Organize the business logic of a service as a collection of DDD aggregates that emit domain events when they created or updated. The service publishes these domain events so that they can be consumed by other services.

Figure 8 Microservices – Domain event

Event Sourcing

As we learned, CQRS and Saga. In CQRS, a command needs to update the database and publish the message so that other services and perform the required action. So the database update and sending a message must be atomic in order to avoid data inconsistency. Event sourcing talks about how reliably and automatically update the database and publish the message/event.

Forces

Services must be loosely coupled so that they can be developed, deployed and scaled independently
Update the database and sending message must be atomic to avoid data inconsistency

Solution

Use the event sourcing when you are updating the database and publishing a message for that event. Applications persist events in an event store, which is a database of events. The store has an API for adding and retrieving an entity’s events. The event store also behaves like a message broker. It provides an API that enables services to subscribe to events. When a service saves an event in the event store, it is delivered to all interested subscribers.

Some entities, such as a Customer, can have many events. In order to optimize loading, an application can periodically save a snapshot of an entity’s current state. To reconstruct the current state, the application finds the most recent snapshot and the events that have occurred since that snapshot. As a result, there are fewer events to replay.

Example

Taking example of Booking and Passenger service. Let’s develop it using event sourcing and CQRS.

Figure 9 Microservices – Event Sourcing

Benefits

It solves one of the key problems in implementing an event-driven architecture and makes it possible to reliably publish events whenever state changes.
Because it persists events rather than domain objects, it mostly avoids the object relational impedance mismatch problem.
It provides a 100% reliable audit log of the changes made to a business entity
It makes it possible to implement temporal queries that determine the state of an entity at any point in time.
Event sourcing-based business logic consists of loosely coupled business entities that exchange events. This makes it a lot easier to migrate from a monolithic application to a microservice architecture.

Drawbacks

It is a different and unfamiliar style of programming and so there is a learning curve.
The event store is difficult to query since it requires typical queries to reconstruct the state of the business entities. That is likely to be complex and inefficient. As a result, the application must use Command Query Responsibility Segregation (CQRS) to implement queries. This in turn means that applications must handle eventually consistent data.

Conclusion

When we move from monolithic to Microservices architecture, we address many challenges like scalability, agility, flexibility, and many more. But when using this architecture there are numerous issues that we must address. When we work on some business cases we need to play around with the data, if our solution architecture is not planned with design patterns instead of getting the benefits of Microservices we will loop into the various issues. So, understand these design patterns and take the maximum benefits of Microservices architecture.

References

Microservice Architecture – https://microservices.io/

Explore more at Teknonauts.com