Wednesday, December 17, 2014

Coding in the cloud

I have found myself using a number of different computers both on the move and at home. This is a pain. The Chrome browser has helped me bring together bookmarks and browsing history across my devices - it has worked flawlessly so far - except on a 7 inch tablet where the tabs are a little too small and the keyboard too large. But what I have wanted for a little while now is a development environment that can be accessed from any machine and anywhere.

There are of course the hosted full blooded virtual instances in Amazons and Microsoft's cloud. So far the MS Azure instances look to be the easiest to use. Remote desktop access may be blocked from certain locations. Another gotch with Azure is that if you select an Azure VM with a paid for product such as Oracle then your account will be blocked the next day and you will have to drop out of the free trail period to continue. AWS looks a little more complex to set up. Watch this space. There are web based coding environments such the Koding web based IDE using Python. See this earlier post. But these are only point solutions.

I spent some time selecting the 'best' Chromebook for me to use as a portable machine, I can not afford a MacBook Air, and small and light Windows machines are either too expensive or not worth the bother. I eventually settled on a Acer C720 I am very happy with this purchase. It is almost perfect for use on the train. The network teathering to a phone works well. So far EE have proved to provided better and faster coverage than giffgaff (O2) on the great western mainline out of Reading.
Next I'll look at the ssh client and Crouton.

Thursday, December 4, 2014

git - Version control

git , implemented as a service by gitlab and github, offers DVCS - distributed version control - which has a number of advantages over previous approaches using CVS, SVN or older proprietary approaches such as PVCS. Gitlab and Github differ mainly in there focus with Gitlab aimed at corporate's that might want continuous integration features 'out-of-the-box'. Github is bigger and has recently been acquired by Microsoft.

Git's basic advantages are that it:
- allows developers to version locally, although I am currently using onedrive to backup active files and folders
- allows sharing, backup and version-ing to the cloud via github
- provides a fast and efficient toolset
- it is either free or very cheap, depending on how it is used.

These advantages are better described here

github is free for a single user, if you don't mind anyone seeing the contents of your repository.

If you understand the principals of version control then this text will bring you up to speed with the product and the key differences with SVN.

Then try the on line interactive tutorial here.

If you understood that the next step I would recommend, assuming you have some basic Linux command line knowledge and are on a Windows machine, is to create a free github account and install git locally within Cgywin. Installing and using git this way does not impact the rest of your Windows installation like using an extension to Windows Explorer would. It also installs everything you need in one process.

For Ubuntu install using sudo apt-get install git-core

If you do not have a github account create one here. My example is 'https://github.com/colfraser/'

Install Cygwin from the project page. Follow the install defaults until you get to the Select Packages dialogue . Type 'git' in the search field and install from the devel category. The package is call 'git: Distributed version control system' or something close to it. In a similar fashion also install ssh, take this from the net category, it will be called something like openssh. The current version of Cgywin references git version 1.7.9 which works fine. The installation may take up to 5 minutes depending on connection speeds.

Check the install at the Cgywin command line with the following command,

git --version

#set up local variables to match how you have set up github.
git config --global user.name "Colin Fraser"

git config --global user.email "address@domain"

#Check the config just entered
more ~/.gitconfig

#This commands generates the security keys which will will ignore for the moment
ssh-keygen -t rsa

create a new repository on github, for example PET_Names. Use the '+' button the top of the screen. Check the box next to 'Initialize this repository with a README'.

env GIT_SSL_NO_VERIFY=true git clone https://github.com/colfraser/PET_Names
#This command allows the SSL security to be ignored so it is not recommended for normal use but it allows the remote repository that we have just created to be instantiated on our local machine.

cd PET_Names # Move into the folder created in the clone process

#Add some files and change the readme
echo 'new line 04/12/2014' >> README.md

git status
# shows modified files

echo "new file" > another_file.txt


git status
# shows modified file and new files

git add .
#add the files to the local 'staging area'

git commit
#commit those files to the local git repository. Add comment in the vi editor box that will pop up. Type 'I' to insert, ':wq' to save.

git status
#will show nothing to commit

git push
#This uploads the committed changes in the local repository to the github repository that was cloned locally previously. The github account name and password will need to be entered. These entries are not echoed to the screen.

#change the xml file using vi or via Windows.

git diff
#will show changes

git add *.xml
git commit
git push

There is a good overview of getting started with git and the GUI local git repository tool tortoise here.

This tool provides an extension to the Windows explorer and was inspired by the SVN tool.

Git for Windows provides a Windows App and 'Git Bash' a command line emulator to access git on Windows like you would on Linux.

A cheat sheet for command line git is here

Friday, September 26, 2014

Useful *nix commands



to update

dbo.Transaction
truncate table dbo.Approval;
dbo.NextTable


to

truncate table dbo.Transaction;
truncate table dbo.Approval;
truncate table dbo.NextTable;


use this in Bash, both lines add text in the appropriate places if not already present

sed -e :a -e '/;$/!s/$/;/' file-list.sql > new-file-list.sql
sed -e :a -e '/truncate table /!s/^/truncate table /' new-file-list.sql > new2-file-list.sql

Show non ascii characters

nonascii() { LANG=C grep --color=always '[^ -~]\+'; }

printf 'ŨTF8\n' | nonascii


To remove line feeds, quotes and substitute quotes for a unique string. Used to format a field containing email content from a MySQL query extracted using concat('#K#',k.a_subject,'#K#') a

cat k2.sql | sed -e 's|["'\'']||g; s/`//g; s/#K#/\x27/g; s/\\n//g' > out.sql

Check for 8 digit numbers

grep -v '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' source.csv > without_8_digits.csv

grep '[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' source.csv > with_8.csv

Extract lines in .csv with occurrences of the strings in .txt

grep -F -f client_people.txt source.csv > client_people_source.csv

sed-oneliners.

Wednesday, May 21, 2014

Online Tools for codeclub

It appears that most of the CodeClub material can be practiced in the Cloud as it were, without having any particular software installed locally other than a modern browser.

Scratch has had an entirely web hosted version since May 2003.

For creating web sites they mention Thimble on the resources page. This brilliant tool can be used for most of the tutorials. It appears to struggle a bit with the exercise that uses CSS, 6 - Hide the Ninja's. The images supplied in the project materials will need to be uploaded somewhere accessible.  Amend the URL's to pickup the image location and amend the street corner declaration block to set the background to not repeat. Place the images in a location that does not result in a 'security - mixed content' error. Placing the images in a SquareSpace gallery worked for me as follows [UPDATE - gallery expired].

  background:url(https://static.squarespace.com/static/537c7f23e4b0b9e2ae500c6b/537c7ff4e4b04863dea97e1b/537c932ce4b0682a2ca2d5bc/1400673081445/street_corner.png?format=500w);   position: absolute;
  background-repeat: no-repeat;

This gist shows some of the other basic HTML syntax that is demonstrated, just paste it into the Thimble window.

For the Python projects try Sculpt. This is an entirely in-browser implementation of Python written using JavaScript - it is a very cool thing. For the turtle examples you will need to actually declare the turtle object.

import turtle
t = turtle.Turtle()
t.forward(30)

There are other Python editors available over the web, for example pythonanywhere and Nitrous.IO,  but they don't appear to handle the Turtle library. If anyone can get these to work I'd be interested to know how.


[Update] Perhaps a bit advanced for the average 10 year old but there is an in-browser implementation of UNIX here. Perfect for learning a little Emacs

Wednesday, May 14, 2014

Primary School Computing

I have had an interest in this for some time as my kids have been getting bigger. For a number of years the primary schools computing curriculum had a focus around ICT, learning Word and Excel and the like, and not programming. In next years curriculum the focus is moving to problem solving and creation of 'dynamic' content and away from just the usage of technology products. I think this will encourage much more involvement and understanding.

I do have some concerns that small schools will struggle to deliver the full breadth of the new curriculum. Teachers are very busy and rightly focus on the basics. Finding the time to develop the skills need to deliver this teaching and arrange the provision of the necessary resources will be a stretch for many.



The curriculum includes the safe use of computing resources, how to gather and organize  information as well as programming. It is not prescriptive in these areas - which could lead to a variety of approaches with different levels of effectiveness.  The codeclub approach to the programming elements looks very useful. For example for 9 year olds the resources cover the graphical Scratch programming tool, website creation and a simple text based programming language called Python (Python skills can later be used on the Raspberry Pi). 



After a two year pause while the new curriculum has been put together some schools will have neglected their hardware while others I know have made big investments in tablets and PCs. Some of the approaches to provisioning hardware have costs which are not always apparent at the start. Desktops need a lot of management and even the best 'Apps' will have limits and costs. The Chromebook approach might be the most cost effective route for many schools. The machines are less expensive to purchase, the management much easier and the resources free or inexpensive. The 'Cloud' as a development space is becoming more capable and simple. The current version of Scratch is best accessed in its web based version. Python & Ruby can be learnt easily in web hosted ways. 


I have not tried the Raspberry PI yet. While it is an inspirational device I think it has some drawbacks for teaching programming in an educational setting. The fact that a separate screen and keyboard and in some instances a protective case are required is a problem. This is a lot of clutter. Some schools may want to re-cycle old peripherals: it should be noted that there is no VGA connector on the PI so a separate converter would be required to use an old monitor. If very old keyboards are being used than a PS/2 to USB adapter may also be required.



Wednesday, March 26, 2014

Excel vLookup vs SQL Join

This will sound rather obvious to Excel wizards. We had a problem with a very complex report that temporarily needed some extra details added. Rather than create a copy of the report we added some extra data to it via Excel. We extracted the extra data via SQL and then joined the two outputs via Excel using vLookup. An example of how to do this is here. The index match lookup method is recomended for very large spreadsheets.


An alternative would have been to pull the data back into the database using Cgywin via a temporary table. This would allow us to do a join via SQL in the usual fashion.

grep 'AP Invoices' ./ap_bal.csv > ap_edited.csv

Create Table Analysis (Transaction_Type Varchar2(130), Transaction_Number Varchar2(20)…………. );

sed 's+^+Insert Into Analysis (Transaction_Type, Transaction_Number) Values (+' ap2.csv > ap3.txt sed 's/$/,)/' ap3.txt > ap4.txt

Clean up and load with SQL Developer

Thursday, March 13, 2014

Oracle EBS Development and Extensibility Handbook

If you are starting to work on Oracle EBS applications this book is the best place to start. I wish I had got hold of a copy sooner.

By the same author is Oracle Fusion Applications Development and Extensibility Handbook (Oracle Press).


Searching and reviewing the SVN repository

I wanted a way to search a subversion repository for change references. I'm using cygwin to run it from Windows. Useful if you are not sure where a change has been made and do not want to create and search a local replica.



To view deletions and modifications of files try the following:

svn log https://vmsscdtconfmgt/SSC_Releases -v --limit 300 | grep 'D /' > svn_del_add.txt

svn log https://vmsscdtconfmgt/SSC_Releases -v --limit 300 | grep 'M /' >> svn_del_add.txt

Wednesday, January 29, 2014

Workday

I have been having a close look at Workday.

UPDATE: This webcast is a bit waffly but is a good primer on Workday. There is a discussion around the absence of customization at the 35 minute mark. And doing no customization at 40 minute mark (experience and outcomes) and integration tools, role of IT dept. and governance at 51 min.

UPDATE: There are Workday services in the government CloudStore.

Some detailed information about the product set is available on the internet but these are the questions I have not found answers to. Anyone know the answers?

• What degree of ‘personalisation/configuration’ is possible - I'm guessing that page fields can be hidden, re-labelled, calculated etc. What else is possible?

  Answer - some columns can be added (reporting issues),
• How do you migrate changes between instances (meta data/config/master data/other setup)?
• What is the toolset like? Fat like PStools/jDeveloper? How do you debug or is it a ‘blackbox’?
   Answer - All Web apart from the integration tools.
• How is reporting facilitated. At the pre-defined object level?
• How are local legal changes catered for, i.e. VAT and Payroll?
• As there is no on-premise option how do you ‘order up’ a test/dev instance? Is it Vanilla? Can you automate a migration process from, say, another dev instance?
Answer - It appears you get one test/dev instance: http://www.kloud-consulting.com/your-organisations-workday-deployment-how-to-build-an-enduring-relationship/
• How do you integrate local or 3rd party solutions, such as say Bottomline payments solutions, just how far does the ‘Cloud integration platform’ go. Do you need another SOA system?
Answer - The Cape Clear base tools are good and getting better
• How do you keep integration's in sync with WDs twice yearly upgrade cycle. What breaks? How do you control the cut-over (other than version-ed web services)?
• How do you\can you set-up multi-currency, multi-language, multi-book, multi-BU etc?
• Can you get down and ‘see’ the data in the noSQL value pair DB? What is the lowest level of interrogation? Do you only look at what comes out of the web services or can you dig deeper.?


After watching a technical webcast it appears that

• There is no access to the underlying database from the applications.

• All data held in the memory. Data saved to persistence store and DR site. No access to the store.

• SaaS updates x2 year. Not x2 a decade.

• Native iPad application – zero cashing

• Looks like the config is done on the browser. No customisations only configurations.

• Enterprise integration builder – For simple single entity integrations ,simple manipulations.

• Workday Studio – Eclipse based framework (fat) developer level tool – to roll your own integrations rather than use the delivered ones or other pre built examples. Workday Studio provides full SOA tool orchestration etc.

• Modern search based UI – like Windows 8 or Ubuntu Unity

• DR sites are full replica's.

References

http://aboutworkday.blogspot.co.uk/2013/08/moving-from-peoplesoft-to-workday.html http://www.hrlab.com/workday-technology-review.php https://community.workday.com/node/3539#hcm-ldap https://community.workday.com/book/export/html/117
https://forms.workday.com/uk/landing_page/webinar_the_register_simplifying_it_with_cloud_lp.php?camp=70180000000poPY&campid=ukhm_fma_no_de_14.1123

Blogging

Why blog?

For me? Well to clarify and expand my thoughts. To trigger discussion. To share the 20-30 sets of work related notes that might be worth sharing.

Technical Blogging

Oracle EBS Technology Review [Draft]

When I started supporting Oracle EBS I could find little summary information that answered technical questions. So I thought I’d try and capture the technical essence of the product and with hindsight answer some of the questions I originally had. I aim to do this over just a couple of pages.

Oracle E-Business Suite is a comprehensive collection of enterprise resource planning, customer relationship management and supply chain management software. This functionality is split into various modules integrated to a greater or lesser degree on a single platform. The product is very widely implemented across thousands of organisations in many counties. Viewed as somewhat legacy in nature Oracle is continuing to extend and maintain the product. The latest version, 12.2, contains technology from, and a path towards, Oracle's leading edge technology ‘Fusion’. Unlike some of the application suites that Oracle has acquired over the last 10 years it continues to successfully sell EBS.

The Oracle applications division was foundered in 1987 and the software that forms EBS originates from that time. The different parts of EBS have been created and extended by teams across the world ever since then. It would be interesting to know what the oldest technical elements are of the current release. But the main issue from a support perspective is that different parts of the application will have been implemented using different tools and technologies. They have evolved using different design approaches and standards. For example some of the core HRMS modules have been written using PL/SQL and Forms and have a common debugging framework that experienced technical supporters will be able to use to help diagnose problems. Other modules such as finance have evolved over a similar time period but code elements are not built using the same approaches and debugging framework as HRMS. The self service parts of HRMS were written many years later using evolving technologies particularly in the user interface elements.

It is very much a suite of applications and technologies rather than an integrated product and unlike some other enterprise software it can only be run on an Oracle database. There are a number of advantages of being tightly integrated into a database. The main one being that code can execute within the database. This can lead to massive performance improvements compared with transferring data within a client server architecture. A key drawback is vendor lock-in and a proprietary drift away from standards, an example being SQL syntax changes.

Oracle forms runtime discussion here. Master data, transaction data, meta data, configuration data

Large teams comprising people with many different individual specialties are required to implement and support the product. This is one reason for the often long turn around times for changes on such projects. Another is often the lack of knowledge by test teams and approvers of the detailed nature of such changes.

It has evolved and dragged a lot of technology with it since 1987. It’s successful because it is successful with wide adoption across many different types of medium and large scales enterprises. There is a large community of literate users and technical supporters. Oracle is a very healthy company with a presence in many countries. It is a huge product, one person can only know a small subset of functionality or technology. In comparison to PeoplesSoft where over the course of 3 or 4 years it might be possible to have an in-depth understanding of the technology and a sufficient grasp of various functional modules to be able to resolve most problems in a time efficient manor – at least with HRMS.
There is no bundled debugger Support difficulties. Greater emphasis on configuration, via flex fields, fast formula (PL/SQL type code triggered from pre configured points in the systems) rather than customisation. For example in payroll.

The Payroll module is a closed box. In fact only a very people in Oracle fully understand how it works. Oracle does not allow more than two of these people to share the same aeroplane.

One of the first things that will surprise a first time ‘power-user’ or technical developer are the two different ways of interacting with the application from their Office PC. For the most part application functionality is accessed via one of two types of clients. The professional client – which is a ‘fat’ client environment and a thinner html based client. This can result in a some what jarring application experience. For example most of the screens used to hire a new employee are written in Forms but at the point where you enter a new salary a HTML page is spawned that fills the screen. Data entered on this page must be saved separately from the data saved in the forms based page.

The HTML client is delivered by OAF. This is a Java based framework that delivers all the security, a standard look and feel, etc, The tool used to produce and extend this is Oracle's Swiss Army knife JDeveloper.

to be continued.

Visual Management - Skills Matrix

It's important in large technical teams that have diverse skill sets to publicise and celebrate individual skills. All teams have limited resources and need to identify where there are coverage gaps and where training time and effort are best directed. Here are some ideas in this area.

Employee visual management. Put visible targets on the wall. Share knowledge of the teams skills.

Get staff to self assess their skills and share teams knowledge graphically as in this example (hat tip JB).

Technical Skills Visual

Use a proven modelling approach that has a definition of skills levels. Such as the Dreysus model of skill acquisition. After initial self assessment get people together to moderate and refine the information.



Technical Skills Matrix Spreadsheet Example - xlsx

Technical Skills Matrix Example - jpg

To help sustain training efforts keep a team Victory log ( hat tip MH) – containing details of successes outside of the normal day to day activities. To record progress within a team and as a way of influencing positive behaviors across a team.

Here are some examples:

12-Jan-2012: ITIL v3 Foundation exam passed
19-Jan-2012. Volunteered at the Feedback session to represent department members at SMT meeting.
3-Feb-2012. Created SQL archive and guidance
06-Mar-12. Recognition from RE for actionable contribution at the SMT meeting.
26-Mar-12. Quote from VA 'X's understanding of architecture and integration is vital to the work that we are doing. He has provided us with exhaustive list of non functional requirements that should enable xxx project to deliver measurable improvements on ERP interfaces as well as overall customer satisfaction.'
26-Mar-12. Challenged poor quality transition team builds. Builds were re-worked
26-Mar-12. Passed x Exam and used these skills on this project