Web scraping for Weather information

This is the first time I use python to scrape web information. I want to write some experiences for that.

 1. About Regular Expression:
   [^>]*: anything in which begins with font;
   \(: use back dash \ following any symbol you want to match (i.e., here ( );
  [^\n]*: means to change to a new line at the end of a line;
  [\w][\w] or [\w]{1,2}: means to match one or two words;
  [\w]* or [\w]+: means >=0 or >=1;
  ([A-Z][A-Z]?): one or two capital letters;
  [\d]: for digitals;
  | : means or ;

  [\d\.]+: for float digitals;

 2. About Web Scraping:
    (1) open URL –> urllib.urlopen(url);
    (2) get source code of URL –> sock.read();
    (3) localise what you want to scrape through regular expression
             –> matcher=re.compile(RE)
                   elements=mather.search(htmlSource) or findall(htmlSource)
                   [search: returns one element;
                     findall: returns a list of results;]
   (4) read results 
           –> for x, y in enumerate(elements):

                 [x -> index of the list; y -> the value]

 3. The whole program:

leave so many things undone!

Now I feel so strange: why do I leave so many things undone when I go back?

1. not finished my paper;

2. not write enough application reports on pervasive computing;

3. not finish the category book, and say nothing of ideas;

4. screen-scraping on weather;

5. cvs trigger (it seems Radu did that himself );

6. do not learn how to use Linux enough;

…..

Before now, I even have not realized the severity of the problem. Now it’s late to be aware or not?

I do not know, but I think at least I need write a report to Simon even though he did not ask me to do.

So depreseed and disapponted~~

OCaml

Caml is a general-purpose programming language, designed with program safety and reliability in mind. It is very expressive, yet easy to learn and use.

It is the first time I learn OCaml: http://caml.inria.fr/pub/docs/manual-ocaml/index.html (Part 1: the core language)
Programming:

1. List
define a list -> let l =["a "; "b";"c"];;
add an element to a list -> “life” ::” l”;;
define a function on list:
– match list with [] -> [] or [elt]
– define the core function: | head :: tail -> function body.
list.assoc returns the data associated with a given key in a list of (key, data) pairs, and raises the predefined exception Not_found when teh key does not appear in the list:

let name_of_binary_digit digit =
try
List.assoc digit [0, "zero"; 1, "one"]
with Not_found -> “not a binary digit”;;

2. Function
let deriv f dx = function x -> ( f ( x +. dx) -. f(x)) /.dx;;
let compose f g = function x -> f (g (x));;
apply function n to each element in a list:
List.map (function n -> n*2+1) [0;1;2;3], or
let rec map f l=
match l with
[] ->[]
| hd::tl -> f hd:: map f tl;;

3. Records
Define a record type:
type ratio = { num: int; denum: int};;
Define a function on record:
let add_ratio r1 r2 =
{num = r1.num * r2.denum + r2.num * r1.denum;
denum = r1.denum * r2.denum};;
Apply the function:
add_ratio {num=1; denum=3} {num=2; denum=5};;

4. The most common usage of variant types is to describe recursive data structures.
Define a binary tree:
type ‘a btree = Empty | Node of ‘a * ‘a btree * ‘a btree;;
Define a recursive function on a binary tree:
let rec member x btree =
match btree with
Empty -> false
| Node(y, left, right) ->
if x = y then true else
if x

5. For:
let add_vect v1 v2 =
let len = min (Array.length v1) (Array.length v2) in
let res = Array.create len 0.0 in
for i = 0 to len – 1 do
res.(i)
apply this function: add_vect [ | 1.0; 2.0 |] [| 2.0; 3.0|];;

6. The let binding is not an assignment, but introduces a new identifier with a new scope. However, the standard library provides references, which are mutable indirection cells, with operators ! to fetch the current contents of the refence and := to assign the contents.
let insertion_sort a =
for i = 1 to Array.length a – 1 do
let val_i = a.(i) in
let j = ref i in
while !j > 0 && val_i

7. Try … with
The with part is actually a regular pattern-matching on the exception value.
let temporarily_set_reference ref newval funct=
let oldval = !ref in
try
ref := newval;
let res = funct () in
ref := oldval;
res
with x ->
ref := oldval;
raise x;;

8. some examples I like:

Example1:

# let rec eval env exp =

match exp with
Const c -> c
| Var v ->
(try List.assoc v env with Not_found -> raise(Unbound_variable v))
| Sum(f, g) -> eval env f +. eval env g
| Diff(f, g) -> eval env f -. eval env g
| Prod(f, g) -> eval env f *. eval env g
| Quot(f, g) -> eval env f /. eval env g;;

# eval [("x", 1.0); ("y", 3.14)] (Prod(Sum(Var “x”, Const 2.0), Var “y”));;

Example2:

# let rec deriv exp dv =
match exp with
Const c -> Const 0.0
| Var v -> if v = dv then Const 1.0 else Const 0.0
| Sum(f, g) -> Sum(deriv f dv, deriv g dv)
| Diff(f, g) -> Diff(deriv f dv, deriv g dv)
| Prod(f, g) -> Sum(Prod(f, deriv g dv), Prod(deriv f dv, g))
| Quot(f, g) -> Quot(Diff(Prod(deriv f dv, g), Prod(f, deriv g dv)),
Prod(g, g));;

# deriv (Quot(Const 1.0, Var “x”)) “x”;;
- : expression =
Quot (Diff (Prod (Const 0., Var “x”), Prod (Const 1., Const 1.)),
Prod (Var “x”, Var “x”))

Python

Python is a clear and powerful OOPL, comparable to Perl, Tcl, Scheme, or Java.

Some features I am interested in:

1. data types are strongly and dynamically typed. Mixing incompatible types causes an exception to be raised.

2. It contains advanced programming features such as generators and list comprehensions.

3. Automatic garbage collection.

4. an intepreted language, which can save considerable time during program development because no compilation and linking is necessary.

Learn Python:

1. NUMBER: complex numbers with a nonzero real components are written as “(real+imagj)”, or can be created with the “complex (real, imag)” function. There is no one correct way to convert a complex number to a real number. The last printed expression is assigned to the variable _.

2. STRING: string[x:y] means string from x to y-1. Assigning to an indexed position in the string results in an error. Indices may be negative numbers, to start counting from the right.

Technical report for Pervasive computing applications

This afternoon, Adrian, Stephen and I discussed something about how to write a technical report for current pervasive computing applications. We will read some papers and find interesting applications. Summarize what we read and post on Twiki pages: http://secure.ucd.ie/twiki/bin/viewauth/SRGWeb/PervasiveCaseStudies

I prepare to do that after my exam on 17th, Dec. It seems to be many things.

1. For technical report: read papers and find applications;

2. For category theory: read books and papers, and think of new ideas;

3. For my paper: finish the last report and convert it into a paper;

4. For programming: learn ML, Haskell, Python, Ocaml, and so on.

ML

These two days I learn how to program with ML that is a functional programming language, quite different from what I have known.

Data types supported: int, real, bool, char #”a”, string “a”, list[, , , ] “::, nil, @”, tuple().

Two ways for programming:

(1) binding symbols: val ……;

(2) define functions: fun …….
1. simple function definitions:

fun function_name parameter = expression;
2. A binding allows us to refer an item to a symbolic name.

val symbolic_name = expression;

3. While a tuple() allows its components to be of mixed types and is of fixed length, a list[] must have identical typed components and may be of any length.
4. A function of more than one argument may be implemented as a function of a tuple or a “curried” function. (After H B Curry).

fun add(x,y)=x+y:int;

fun add x y = x+y:int; The type of this function is int->(int->int). It is a function which takes an integer and returns a function from an integer to an integer.

5. About list:

Usually a list is expressed as h::t, where h is the first element of the list. The common operators are :: and @, both of which are used to construct functions.

6. Map

fun map f nil = nil

| map f (h::t) = (f h)::(map f t);

For example, val doublist= map (fn x=>2*x);

7. Reduce takes a binary function (a function with two inputs) a base value and a list. It applys the function repeatedly down the list.
fun reduce f b nil = b

| reduce f b (h::t) = f(h,reduce f b t);
For example, val flatten = reduce (fn(a,b)=>a@b) nil;

8. Zip can be used to apply a binary function (one with two inputs) to the corresponding elements of two lists.

fun zip f nil nil = nil

| zip f (h::t) (i::u) = f(h, i) ::zip f t u;

9. Filter takes a predicate (a function which returns true or false) and a list. It returns the list with only those items for which the predicate s true.

fun filter f nil = nil

| filter f (h::t) = if f h then h::filter f t else filter f t;

10. other common functions:

fun member x nil = false

| member x (h::t) = x=h orelse member x t;

————————————————

fun fst(a,_) = a; (* Also try #1 *)

————————————————
fun snd(_,b) = b; (* Try #2 *)

10. Nested definitions: let … in … end

fun sort nil = nil : int list

| sort(h::t) = let

fun insert(i,nil) = [i]

| insert(i,h::t) = if i > h then i::h::t else h::insert(i,t)

in insert(h, sort t) end;

——————————————————————–

fun rev l = let

fun reva(nil,acc) = acc

| reva(h::t,acc) = reva(t,h::acc)

in reva(l,nil) end;

—————————————————————————
fun power(x,0) = 1

| power(x,n) = let

fun even n = (n mod 2) = 0 val s = power(x, n div 2)

in if even x then s*s else x*s*s end;

—————————————————————–

fun minmax [x] = (x, x)

| minmax(h::t) = let

val (mn, mx) = minmax t

in (min(h,mn),max(h,mx)) end;

Karen Henricksen

 2006

1.      Developing context-aware pervasive computing applications: models and approach (2006)

This paper addresses the software engineering process, including context modeling techniques, a preference model for representing context-dependent requirements, and tow programming models.

2005

 

2.      Middleware for distributed context-aware systems (2005)

Most current middleware or infrastructure for context-aware systems have not adequately addressed issues such as mobility, fault tolerance or privacy. This paper provides an analysis of the requirements of a middleware, and a critical review of several middleware solutions and their own PACE middleware.

3.      Personalizing context-aware applications (2005)

This paper characterizes several approaches to personalization of context-aware applications and introduce our research on personalization using a novel preference model.

4.      Modeling context information with ORM （2005）

This paper addresses, drawing on their experiences with using an ORM-based context modeling approach to develop a variety of context-aware applications. It also shows how their extended variant of ORM is used to support the development of context-aware applications, and outline some of the remaining challenges.

5.      Context obfuscation for privacy via ontological descriptions (2005)

The paper proposes privacy protection mechanisms for users, which are intended to provent breaches of user privacy through unauthorized context disclosure. It describes a new obfuscation mechanism that can adjust the granularity of different types of context information to meet disclosure requirements stated by the owner of the context information.

6.      Applying a disciplined approach to the development of a context-aware communication application (2005) 

This paper demonstrates the use of a disciplined, model-based approach to engineer a context, session initiation protocol based communication application. The approach enables the description, acquisition, management and exploitation of arbitrary types of context and user preference information to enable adaptation to context changes. We consider a self-adapting communication application that exploits context and preference information to allow users to communicate seamlessly with one another.

7.      Extending context models for privacy in pervasive computing environments （2005）

This paper tackles the ownership challenges in an attempt to provide one of the missing pieces required for a complete privacy one of the missing pieces required for a complete privacy solution for context-aware systems. It argues that ownership information forms a natural extension to context models, and propose the integration of flexible notions of ownership into their previously developed context modeling techniques.

2004

8.      Towards a common context model for virtual community applications (2004)

This paper describes how the complexity associated with designing and implementing context-aware applications can be reduced through both context reuse and programming methods that allow context evaluation to be decoupled from applications.

9.      Towards a hybrid approach to context modeling, reasoning and interoperation

This paper investigates evaluates the most appropriate uses of ontology languages and tools in context-aware systems, and to explore the creation of a new hybrid solution that combines ontology concepts with previously developed approach to context modeling and reasoning.

10.  Automating context-aware application development

This paper describes an extensible set of tools they are developing for use in conjunction with their existing context modeling framework and infrastructure that provides substantial assistance to the development and deployment of context-aware applications.

11.  A software engineering framework for context-aware pervasive computing

This paper presents a conceptual framework and software infrastructure that together addresses known software engineering challenges, and enables further practical exploration of social and usable issues by facilitating the prototyping and fine-tuning of context-ware applications.

12.  Modeling and using imperfect context information

This paper explores the problem of imperfect context information and some of its causes, and proposes a novel approach for modeling incomplete and in accurate information.

2003

13.  Scalable location management for context-aware systems (2003)

This paper presents a location management system able to gather process and manage location information from a variety of physical and virtual location sensors. The system scales to the complexity of context-aware applications, to a variety of types and large number of location sensors and clients, and to geographical size of the system.

14.  Generating Context management infrastructure from high level context models

This paper presents a context modeling approach that offers a means for developers to describe and program with context at a high level, without the need to consider issues related to context gathering, management or representation. It also describes a mapping process that transforms high-level context models to management systems capable of maintaining and supplying context information to applications at run-time.

15.  Experiences in Using CC/PP in context-aware systems

The proposed context model is based on the CC/PP standard proposed to support context negotiation between web browsers and servers. They have defined a set of CC/PP components and attributes that allow expressing a variety of context information types and relationships between context descriptions.

2002

16.  Modeling context information in pervasive computing systems(2002)

This paper is concerned with the development of appropriate context modeling concepts for pervasive computing, which can form the basis for such a context management infrastructure. This model overcomes problems associated with previous context models, including their lack of formality and generality, and also tackles issues such as wide variations in information quality, the existence of complex relationships amongst context information and temporal aspects of context.

2001

17.  Infrastructure for pervasive computing: challenges (2001)

This paper presents our vision of pervasive computing and enumerates the software engineering challenges involved in realizing this vision. It also evaluates the current state of research and presents an agenda for future investigations in pervasive computing.