@h12.me

RealTest: real test environment for Go

14 November 2015

h12.me/realtest provides real test environment for Go unit testing, based on Docker.

Includes:

  • Queue
    • Kafka cluster (with zookeeper)
  • Database:
    • MySQL
    • MongoDB
  • Cache
    • Redis
  • Configuration service
    • ZooKeeper

Author

Hǎiliàng Wáng

Contributors

Aliyun OSS Go SDK

22 October 2015

Build
Status Coverage
Status GoDoc

Aliyun OSS (Object Storage Service) Go SDK is a client SDK to access Aliyun OSS API, implemented in the Go programming language.

Installation

go get -u github.com/aliyun/aliyun-oss-go-sdk/oss
go test -v -cover github.com/aliyun/aliyun-oss-go-sdk/oss

Highlights

  • Complete set of Aliyun OSS API
  • Thouroughly tested
    • 100% test coverage
    • intuitive table driven tests
    • full test suite completes within 2 seconds
  • Lint clean
    • golint
    • go fmt
    • goimports
    • go vet
    • race detector
  • Idiomatic & elegant
    • response is returned as a parsed object
    • error is returned as a Go error
    • named options for setting headers & parameters
  • Great extensibility
    • clean and orthogonal implementation
    • users can easily extend the SDK when a new API method is supported
  • No third party dependencies

Documentation

Differences with Python SDK

  • HTTP header User-Agent, e.g. aliyun-sdk-go/0.1.1 (Linux/3.16.0-51-generic/x86_64;go1.5.1)
  • Go HTTP client does not support 100-Continue (will be supported after Go 1.6, see https://github.com/golang/go/issues/3665)
  • HTTP header keys are automatically converted into canonical format, e.g. x-oss-acl becomes X-Oss-Acl
  • Go GET request does not have redundant “Content-Length: 0” header
  • Parameters will be omitted if the argument is not set
  • Go always sends URL parameters and headers in canonical order

Author

Hǎiliàng Wáng

License

licensed under the Apache License 2.0

References

How to Remove All Exited Containers & Dangling Images for Docker?

3 August 2015
docker ps --quiet --filter=status=exited | xargs docker rm
docker images --no-trunc=true --filter="dangling=true" --quiet=true | xargs docker rmi

kpax: a modular & idiomatic Kafka client in Go

23 July 2015

GoDoc

Install

go get -u h12.me/kpax

Design

The client is built on top of Kafka Wire Protocol (i.e. low-level API). The protocol related types & marshal/unmarshal functions are automatically generated by wipro from the HTML spec).

(-) means to be done.

Sub packages

  • model is an abstraction model for request, response, broker and cluster
  • broker is a lazy, asynchronous and recoverable round tripper that talks to a single Kafka broker
  • cluster is a metadata manager that talks to a Kafka cluster
  • proto contains both low level API and a “middle” level facade
  • producer: fault tolerant high-level producer (batching and partitioning strategy)
  • consumer: fault tolerant high-level consumer (consumer group and offset commit)
  • log: replaceable global logger
  • cmd
    • kpax: command line tool to help with Kafka programming

Compatibility

Compatible with Kafka Server 0.8.2.

Error Handling

  • broker
    • fail fast: timeout returns error immediately
    • release resources carefully
    • reconnect when requested next time
  • client
    • metadata reload lazily (only when a leader/coordinator cannot be found in cache)
    • leader/coordinator should be deleted on error
  • producer
    • fail over to another partition
    • failed partition will be retried again after a period of time
    • partition expand (-)
  • consumer
    • just loop & wait on error
    • partition expand (-)
  • graceful shutdown (-)

Efficiency

  • efficiency
    • batching
    • consumer response
    • consumer request (-)
    • producer (-)
    • decompression
    • snappy
    • gzip (-)
    • compression
    • snappy (-)
    • gzip (-)

Author

Hǎiliàng Wáng

Contributors

A Comprehensive Note about Proxy Settings

8 July 2015

Git

> cat ~/.gitconfig
[http]
	proxy = socks5://[host]:[port]
[https]
	proxy = socks5://[host]:[port]

Curl (Homebrew)

> cat ~/.curlrc
socks5 = "[host]:[port]"

Docker pull

env HTTP_PROXY=http://xxx docker pull yyy

GAE

  1. Set proxy

    export HTTP_PROXY http://[host]:[port]
    export HTTPS_PROXY http://[host]:[port]
    
  2. Delete cacerts.txt under GAE SDK installation to disable SSL verification ( make sure the proxy is safe before this step).

  3. Deploy.

    appcfg.py . --email=[address] --passin
    

Phonetic Notations

30 June 2015

Google Dictionary

Google dictionary uses a phonetic notation same as NORD (New Oxford American Dictionary).

To figure out an unknown phonetic notation, the simple way is to look up a few words that you know how to pronounce: church, hat, which, judge, game, thing, ship, thin, this, yes, pleasure cat, pay, care, father, arm, let, see, here, city, my, pot, no, caught, war, force, boy, put, tour, boot, out, cut, word, item, rabbit, winner, pupil

Notes on the Design of Go

29 June 2015

Refusing new features

Avoid leaky abstraction as much as possible. If an abstraction is not solid enough, you’d rather not use them at all. Go is designed like this, the features are carefully selected. If a feature is not solid enough, it will not be allowed to enter Go.

No implicit string concatenation

Rob: That used to be in the language but was dropped when the semicolon insertion rules went in. You need the + to be able to span lines, and if you can’t span lines operatorless concatenation is close to pointless.

No Implicit conversion

Rob: In any case the clarity of Go’s strictness is worth the occasional conversion. A huge class of bugs is simply gone, and a huge piece of tricky language in the specification never needed to be written.

Russ: Inserting automatic widening means giving up the “named types are not the same” rule, which makes it pretty much a non-starter.

Octal literal

Octal numeric system is rarely used, why does Go support them?

From Wikipedia “Octal”: Octal representation of non-ASCII bytes may be particularly handy with UTF-8, where any start byte has octal value \3nn and any continuation byte has octal value \2nn.

Null pointer (billion-dollar mistake)

Jonathan: in the eyes of the Go designers, who clearly don’t view it as a billion-dollar mistake, and in my eyes as well, having spent days debugging (or failing to debug) race conditions, but rarely more than an hour per null-pointer bug).

rog: even in languages with no nil, such as Haskell, you can still get exceptions from using things that have an unexpected form. e.g. head [] that’s not too different from a nil pointer exception. so even if you go the non-nil route, you may still end up paying a good proportion of the “billion dollar” price.

Return errors rather than exceptions

Raymond Chen: It’s easy to write bad code, regardless of the error model. It’s hard to write good error-code-based code since you have to check every error code and think about what you should do when an error occurs. It’s really hard to write good exception-based code since you have to check every single line of code (indeed, every sub-expression) and think about what exceptions it might raise and how your code will react to it. (In C++ it’s not quite so bad because C++ exceptions are raised only at specific points during execution. In C#, exceptions can be raised at any time.)

Nil channel always blocks (both send and receive)

Rob: It’s for consistency with select. The semantics of a nil channel are the same regardless of how it is used. It’s useful that it blocks in a select, so that’s what it does outside a select.

Reading a closed channel always succeeds with zero value

  1. the read cannot block because then it will not be distinguishable with nil channel.
  2. the read cannot panic because the writer close the channel to signal that it has been closed, not to panicking the reader.
  3. Then the read has to return, which value to return? Since the channel has been closed, so the only possible value is a fake one – default zero value. an additional result of type bool reporting whether the communication succeeded. e.g. v, ok := <-ch.

Writing a closed channel panics

The channel should only be closed by the writer(not reader), so if a writer close the channel and then write it again, it should be a programming bug and panic is the right behavior.

Full qualified package name

Rob:

Using . as an import name is part of the language and necessary in some cases to solve difficult naming issues but it’s not the intended style. Package authors can depend on the fact that their code lives in its own name space. If . becomes standard, then all packages in effect become a global name space and the style changes. Instead of

foo.New

the function must be called something like

foo.NewFoo

to avoid colliding with New from other packages. And then what if Foo is itself a common name such as Buffer? You rapidly end up in the Java space of very long names for everything.

This convention we have, using the package names always, is deliberate. Yes, sometimes it makes for more typing but we believe in the long run it means less typing.

Slice type is a builtin type

To support variadic arguments.

len returns int rather than uint

Ian:

Unsigned types are not a good choice for counts, because they have odd behaviour at 0, a common case. Signed types have odd behaviour at very large and very small values, an uncommon case.

What is type T []T

A slice is a small object containing a pointer to the underlying array, so T is the slice object containing a pointer to the underlying array, whose element is T itself.

Each if, for, and switch statement is considered to be in its own implicit block

It means a for loop has two nested blocks, the outer implicit block is for the loop variable, and the inner block for the loop body:

for i := 0; i < 5; i++ {
	fmt.Println("outer i:", i)
	i := 999
	fmt.Println("inner i:", i)
}

The implicit block has to be introduced to for statement so that the the value of the loop variable can be shared between iterations, but I am not sure why it is useful for if and switch.