Author: Bartosz Wieczorek

zookeeper and kafka on vpn changing bind port from 0.0.0.0

me@MacBook:~$ vim update-ip-to-hostname-in-etc-host.sh
#!/bin/bash
ip=$(ifconfig | grep 'inet\ ' | grep -v 127.0.0.1 | awk '{ print $2 }' | tail -n 1)
if [ "$ip" == "" ]; then
ip=127.0.0.1;
fi
echo "Updating IP HOSTNAME mapping to '$ip' '$HOSTNAME' in /etc/hosts ..."
sudo sed -i "" "s/[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\ \ \ $HOSTNAME/$ip\ \ \ $HOSTNAME/g" /etc/hosts
echo "Results for $HOSTNAME:"
cat /etc/hosts | grep $HOSTNAME

Results:

me@MacBook:~$ ./update-ip-to-hostname-in-etc-host.sh  
Updating IP HOSTNAME mapping to '10.162.224.245'   'C02S53DRG8WM' in /etc/hosts ...
Results for C02S53DRG8WM:
#10.162.224.245   C02S53DRG8WM #export SPARK_LOCAL_IP=10.162.209.59 or SPARK_LOCAL_IP="127.0.0.1"
10.162.224.245   C02S53DRG8WM

me@MacBook:~/dev/env/kafka_2.11-0.11.0.0$ vim config/zookeeper.properties
# Added
clientPortAddress=MacBook

Results:

[2017-09-08 16:08:14,726] INFO binding to port C02S53DRG8WM/10.162.224.245:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-09-08 16:08:20,801] INFO Accepted socket connection from /10.162.224.245:50432 (org.apache.zookeeper.server.NIOServerCnxnFactory)
...
[2017-09-08 16:08:20,831] INFO Established session 0x15e61d1d3990000 with negotiated timeout 6000 for client /10.162.224.245:50432 (org.apache.zookeeper.server.ZooKeeperServer)

me@MacBook:~/dev/env/kafka_2.11-0.11.0.0$ vim config/server.properties
#zookeeper.connect=localhost:2181
zookeeper.connect=MacBook:2181

Results:

[2017-09-08 16:08:20,801] INFO Socket connection established to C02S53DRG8WM/10.162.224.245:2181, initiating session (org.apache.zookeeper.ClientCnxn)
...
[2017-09-08 16:08:21,340] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
Advertisements

Remapping winkey windows key when running Windows and Ubuntu in VirtualBox on Mac OSX

If you are used to Mac keyboard and you want to keep using command key for copy, paste, select etc.. also while running Windows or Ubuntu in VirtualBox then you could remap left windows key to left control key by running for:

  1. Ubuntu
    gsettings set org.gnome.desktop.input-sources xkb-options "['altwin:ctrl_win']"
    and optionally set terminal as well
    gsettings set org.gnome.Terminal.Legacy.Keybindings:/org/gnome/terminal/legacy/keybindings/ paste 'v'
  2. Windows
    by adding registry entry:
    Windows Registry Editor Version 5.00
    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
    "Scancode Map"=hex:00,00,00,00,00,00,00,00,02,00,00,00,1d,00,5b,e0,00,00,00,00

    based on https://superuser.com/questions/1190329/can-i-switch-the-alt-and-ctrl-keys-on-my-keyboard

and restarting the Ubuntu or Windows.

xsd schema – repeating elements via unbouned choice and sequence

For the schema below with choice:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="http://www.bawi.com/schema" targetNamespace="http://www.bawi.com/schema" elementFormDefault="qualified">
    <xs:element name="MyRootElement">
        <xs:complexType>

	    <!-- allow either (zero or more of A) or (only exactly single B) -->	
            <xs:choice>
                <xs:element name="A" maxOccurs="unbounded"/>
                <xs:element name="B" />
            </xs:choice>
        </xs:complexType>
    </xs:element>
</xs:schema>

Screen Shot 2017-08-07 at 13.33.10

we get valid xmls:
– no children

<MyRootElement xmlns="http://www.bawi.com/schema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bawi.com/schema schema.xsd">
</MyRootElement>

– zero or more A element(s)

<MyRootElement xmlns="http://www.bawi.com/schema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bawi.com/schema schema.xsd">
  <A></A>
  <A/>
</MyRootElement>

– exactly one B element

<MyRootElement xmlns="http://www.bawi.com/schema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bawi.com/schema schema.xsd">
  <B/>
</MyRootElement>

However, combination of A and B is not possible:

<MyRootElement xmlns="http://www.bawi.com/schema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bawi.com/schema schema.xsd">
  <A></A>
  <B/>
</MyRootElement>

as we get

Invalid content was found starting with element 'B'. One of '{"http://www.bawi.com/schema":A}' is expected.

and also not valid xml:

<MyRootElement xmlns="http://www.bawi.com/schema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bawi.com/schema schema.xsd">
  <B/>
  <A></A>
</MyRootElement>

due to

Invalid content was found starting with element 'A'. No child element is expected at this point.

In order to get xml with many A elements and single B we would define a sequence:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="http://www.bawi.com/schema" targetNamespace="http://www.bawi.com/schema" elementFormDefault="qualified">
    <xs:element name="MyRootElement">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="A" minOccurs="0" maxOccurs="unbounded"/>
                <xs:element name="B" />
            </xs:sequence>
        </xs:complexType>
    </xs:element>
</xs:schema>

Screen Shot 2017-08-07 at 13.47.48

so that the xml below is valid:

<MyRootElement xmlns="http://www.bawi.com/schema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bawi.com/schema schema.xsd">
  <A></A>
  <B/>
</MyRootElement>

For validation I have used the following code:

import org.xml.sax.SAXException;

import javax.xml.XMLConstants;
import javax.xml.transform.Source;
import javax.xml.transform.stream.StreamSource;
import javax.xml.validation.Schema;
import javax.xml.validation.SchemaFactory;
import javax.xml.validation.Validator;
import java.io.File;
import java.io.IOException;

public class MyValidator {
    public static void main(String[] args) throws SAXException, IOException {
        File schemaFile = new File("schema.xsd");
        Source xmlFile = new StreamSource("generated.xml");
        SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
        Schema schema = schemaFactory.newSchema(schemaFile);
        Validator validator = schema.newValidator();
        validator.validate(xmlFile);
        System.out.println(xmlFile.getSystemId() + " is valid");
    }
}

xsd schema declaring xml only with attributes

For schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="http://www.bawi.com/schema" targetNamespace="http://www.bawi.com/schema" elementFormDefault="qualified">
  <xs:element name="MyRootElement">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="MyElement">
          <xs:complexType>
            <xs:attribute name="myAttribute"/>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

Screen Shot 2017-07-27 at 22.06.15
we have valid xml:

<?xml version="1.0" encoding="UTF-8"?>
<MyRootElement xmlns="http://www.bawi.com/schema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bawi.com/schema schema.xsd">
  <MyElement myAttribute="anyType"/>
</MyRootElement>

but when we add text body (e.g. ‘aaa’) to the xml:

<?xml version="1.0" encoding="UTF-8"?>
<MyRootElement xmlns="http://www.bawi.com/schema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bawi.com/schema schema.xsd">
  <MyElement myAttribute="anyType">aaa</MyElement>
</MyRootElement>

then we get validation error:

lineNumber: 3; columnNumber: 51; cvc-complex-type.2.1: Element 'MyElement' must have no character or element information item [children], because the type's content type is empty.

In order to fix it we need to add simepleContent and extension to the schema:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns="http://www.bawi.com/schema" targetNamespace="http://www.bawi.com/schema" elementFormDefault="qualified">
  <xs:element name="MyRootElement">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="MyElement">
          <xs:complexType>
            <xs:simpleContent>
              <xs:extension base="xs:string">
                <xs:attribute name="myAttribute"/>
              </xs:extension>
            </xs:simpleContent>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

Then both xml above with be valid.
Note text-like information in the MyElement icon:
Screen Shot 2017-07-27 at 22.19.43

Empty elements such as

<MyElement/>

is allowed when schema defines it as

<xs:element name="MyElement"/>

or

<xs:element name="MyElement" type="xs:string"/>

. However, when we change schema to int type such as

<xs:element name="MyElement" type="xs:int"/>

then we get validation error:

lineNumber: 3; columnNumber: 15; cvc-datatype-valid.1.2.1: '' is not a valid value for 'integer'.

XSD schema validation of XML

Note the schema and the generated valid xml below (demonstrating different xsd schema syntax features):

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" attributeFormDefault="unqualified">
	<xs:element name="MyRootElement">
		<xs:annotation>
			<xs:documentation>Comment describing your root element</xs:documentation>
		</xs:annotation>
		<xs:complexType>
			<xs:sequence>
				<xs:element name="MySubElement"/>
				<xs:element name="MySubElementString" type="xs:string"/>
				
				<!-- by default elements are required -->
				<xs:element name="MySubElementOptional" minOccurs="0"/>

				<xs:element name="MySubElementRequiring3CharString">
					<xs:simpleType>
					
						<!-- restriction needs to have base type -->
						<xs:restriction base="xs:string">
							<xs:length value="3"/>
						</xs:restriction>
					</xs:simpleType>
				</xs:element>
				
				<xs:element name="MySubElementDisallowingText">
					<xs:complexType>
					</xs:complexType>
				</xs:element>
				
				<xs:element name="MySubElementWithOptionalAttrAndOptionalText" type="MySubElementWithOptionalAttrAndOptionalText" />

			</xs:sequence>
			
			<!-- attribute needs to be below sequence -->
			
			<xs:attribute name="myAttribute" type="xs:integer"/>
			
			<!-- by default attributes are optional -->
			<xs:attribute name="myRequiredAttribute" type="xs:integer" use="required"/>
			
		</xs:complexType>
	</xs:element>
	
	<xs:complexType name="MySubElementWithOptionalAttrAndOptionalText">
		<xs:simpleContent>
			<xs:extension base="xs:string">
				<xs:attribute name="myIntegerAttribute" />
			</xs:extension>
		</xs:simpleContent>
	</xs:complexType>
</xs:schema>

Here is a sample xml (valid to the schema above)

<?xml version="1.0" encoding="UTF-8"?>
<MyRootElement xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" myAttribute="0" myRequiredAttribute="0" xsi:noNamespaceSchemaLocation="my-schema.xsd">
	<MySubElement>text</MySubElement> <!-- can be replaced with: <MySubElement /> -->
	<MySubElementString>String</MySubElementString> <!-- can be replaced with: <MySubElementString /> -->
	<MySubElementOptional>text</MySubElementOptional> <!-- can be completely removed -->
	<MySubElementRequiring3CharString>aaa</MySubElementRequiring3CharString>
	<MySubElementDisallowingText/>
	<MySubElementWithOptionalAttrAndOptionalText myIntegerAttribute="text">String</MySubElementWithOptionalAttrAndOptionalText>
	<!-- can be replaced with: 	<MySubElementWithOptionalAttrAndOptionalText /> -->
</MyRootElement>

my-schema.xsd